JP2018206080A - Head-mounted display device, program, and control method for head-mounted display device - Google Patents

Abstract

To provide a head-mounted display device, a program, and a control method for a head-mounted display device, which increase the accuracy of inputs made by an indication body to improve the usability of a user.SOLUTION: The head-mounted display device includes an image display unit for displaying an image in a display area so as to be transparent to the outside scene and visible together with the outside scene; a camera and imaging control unit 147 for detecting an indication body associated with a motion of a user in an area including at least a range transmitting through the image display unit; and a display control unit 153 for displaying an image showing the detected state of the indication body by the camera and imaging control unit 147 in the display area, and displaying candidates of input contents corresponding to at least one of the motion, shape, and position of the indication body detected by the camera and imaging control unit 147.SELECTED DRAWING: Figure 5

Description

  The present invention relates to a head-mounted display device, a program, and a control method for the head-mounted display device.

2. Description of the Related Art Conventionally, there has been known an apparatus that detects a movement or the like of a pointer and executes processing corresponding to the detected movement or the like of the pointer (see, for example, Patent Document 1).
Patent Document 1 discloses a sign language recognition device that captures a finger character of a signer and processes the captured image data to recognize the finger character of the signer.

JP 2012-252581 A

By the way, a head-mounted display device that is mounted on the user's head and visually recognizes an image superimposed on the outside scene may deteriorate the visibility of the outside scene depending on the displayed image. In some cases, it was difficult to operate the operation unit. For this reason, an input method with high input accuracy and ease of use has been demanded.
An object of the present invention is to improve the accuracy of input by an indicator and improve the usability of a user.

In order to solve the above-described problems, the present invention is a head-mounted display device mounted on a user's head, which displays an image in a display area so that the image can be seen through the outside scene and visible together with the outside scene. And an image showing a detection state in which the detection unit detects the indicator in a region including at least a range that passes through the display unit, and a detection unit that detects the indicator associated with the user's action And a display control unit that displays input content candidates corresponding to at least one of the movement, shape, and position of the indicator detected by the detection unit.
According to the present invention, the image indicating the detection state of the indicator by the detection unit is displayed in the display area. For this reason, it is easy to confirm the detection state of the indicator by the detection unit. In addition, candidates for input content corresponding to any of the movement, shape, and position of the indicator detected by the detection unit are displayed at positions corresponding to images indicating the detection areas. For this reason, the input precision by a pointer can be improved and a user's usability can be improved.

Further, according to the present invention, the display control unit displays an image indicating a detection state of the detection unit corresponding to a position where the indicator is visually recognized in the outside scene.
According to the present invention, since the image indicating the detection state of the detection unit is displayed in correspondence with the position where the indicator is visually recognized, the user can check the detection state while operating the indicator.

Further, according to the present invention, the display control unit corresponds to the candidate corresponding to at least one of the movement, shape, and position of the indicator detected by the detection unit as an image indicating a detection state of the detection unit. Display in position.
According to the present invention, the candidate corresponding to at least one of the movement, shape, and position of the indicator detected by the detection unit is displayed at a position corresponding to the image indicating the detection state of the detection unit. The detection state of the detection unit can be confirmed while operating the body.

Further, according to the present invention, the display control unit displays, in the display area, an image indicating a detection area where the detection unit detects the indicator as an image indicating a detection state of the detection unit. The candidate corresponding to at least one of the movement, shape and position of the indicator to be detected is displayed at a position corresponding to the image indicating the detection area in the display area.
According to the present invention, since an image indicating a detection area in which the detection unit detects the indicator is displayed as an image indicating the detection state of the detection unit, it is easy to confirm the detection area in which the indicator is detected, Operability can be improved. In addition, since a candidate corresponding to at least one of the movement, shape, and position of the indicator detected by the detector is displayed at a position corresponding to the image indicating the detection area, the user operates the indicator while operating the indicator. You can check the detection status.

Further, according to the present invention, the display control unit displays the candidate in a candidate display area of the display area, and detects the candidate display area in a position corresponding to the image indicating the detection area or in the detection area. It is displayed at a position corresponding to the indicated indicator.
According to the present invention, the image indicating the candidate display area is displayed at a position corresponding to the image indicating the detection area or a position corresponding to the indicator detected in the detection area. This simplifies the operation of selecting candidates displayed in the candidate display area.

In the present invention, the display control unit may detect at least one of the detected movement, shape, and position of the indicator when the detection unit detects the indicator at a position corresponding to the candidate display area. Based on the above, one of the plurality of candidates displayed in the candidate display area is selected.
According to the present invention, one of a plurality of candidates can be selected based on at least one of the movement, shape, and position of the indicator.

In the present invention, the display control unit detects the movement of the indicator when the detection unit detects the indicator at a position corresponding to the candidate character displayed in the display area. Based on at least one of shape and position, one of the plurality of candidate characters displayed in the candidate display area is selected.
According to the present invention, one of a plurality of candidate characters can be selected based on the movement, shape, and position of the indicator moved to the position corresponding to the candidate character.

Further, in the present invention, when the display control unit displays a plurality of candidates, and the moving direction of the indicator detected by the detection unit corresponds to any direction in which the candidates are displayed, The corresponding candidate is selected.
According to the present invention, a candidate to be selected can be selected by moving the indicator in a direction in which the candidate to be selected is displayed.

Further, according to the present invention, the display control unit displays the candidate when the shape of the indicator detected by the detection unit is the same shape for a preset time.
According to the present invention, candidates can be displayed by making the shape of the indicator the same shape for a preset time. For this reason, a candidate can be displayed by simple operation.

In the present invention, the display control unit displays the candidate when the position of the indicator detected by the detection unit overlaps a preset position.
According to the present invention, the candidate is displayed when the indicator overlaps a preset position. For this reason, a candidate can be displayed by simple operation with a pointer.

In the present invention, the display control unit displays the candidate when the indicator detected by the detection unit satisfies a preset condition.
According to the present invention, candidates are displayed by setting the shape of the indicator to a shape corresponding to a preset condition. For this reason, a candidate can be displayed by simple operation.

Further, according to the present invention, the display control unit detects, as an image indicating the detection area, an image indicating a detection area for detecting an indicator including the user's left hand and an indicator including the user's right hand. And an image showing a detection area to be displayed.
According to the present invention, an image indicating the detection area of the indicator including the left hand of the user and an image indicating the detection area of the indicator including the right hand are respectively displayed. For this reason, it is not necessary to determine whether the indicator detected by the detection unit is an indicator including the left hand or an indicator including the right hand, reducing the time for displaying the candidate, and the candidate intended by the user Can increase the probability of being displayed.

In the present invention, the detection unit detects a first indicator including the left hand of the user and a second indicator including the right hand of the user as the indicator, and the display control unit includes: A first operation image that is an operation target of the first indicator detected by the detection unit and a second operation image that is an operation target of the second indicator are displayed on the display area.
According to the present invention, it is possible to display the first operation image operated by the first indicator including the user's left hand and the second operation image operated by the second indicator including the user's right hand. it can. For this reason, the operation of the first operation image by the first indicator and the operation of the second operation image by the second indicator can be performed.

The present invention further includes a control information output unit that outputs control information to an external device in response to at least one of the movement, shape, and position of the indicator detected by the detection unit.
According to the present invention, control information corresponding to at least one of the movement, shape, and position of the indicator is output to an external device. Therefore, an external device can be operated by at least one of the movement, shape, and position of the indicator.

The present invention further includes a control information output unit that outputs control information to an external device in response to at least one of the movement, shape, and position of the indicator detected by the detection unit, A first indicator including the user's left hand and a second indicator including the user's right hand are detected as the indicator, and the control information output unit is configured to detect the first instruction detected by the detection unit. The control information including at least one of first information corresponding to the operation of the body and second information corresponding to the operation of the second indicator detected by the detection unit is output.
According to the present invention, the control information including at least one of the first information detection corresponding to the operation of the first indicator detected by the detection unit and the second information corresponding to the operation of the second indicator is the control information. Output by the output unit. For this reason, it becomes possible to operate an external apparatus by operation of a 1st indicator, or operation of a 2nd indicator.

The present invention further includes a voice input unit for inputting voice, and the detection unit detects the indicator when the voice input unit inputs a preset voice.
According to the present invention, it is possible to cause the head-mounted display device to start detecting an indicator by using a preset voice.

The present invention further includes an operation detection unit that detects an operation, and the detection unit shifts to a state in which the indicator is detected when an operation preset by the operation detection unit is detected.
According to the present invention, the head-mounted display device can be shifted to a state in which the indicator is detected by performing a preset operation.

Further, the present invention provides a gesture registration dictionary in which a gesture indicating a preset movement trajectory of the user's finger as the indicator, the candidate is associated, and a preset shape of the finger. A storage unit that stores at least one of a hand sign shown and a hand sign registration dictionary in which the candidate is associated, and the display control unit includes the movement or shape of the finger detected by the detection unit, and the gesture The gesture or the handsign is selected by matching with a registration dictionary or the handsign registration dictionary, and the candidate corresponding to the selected gesture or the handsign is displayed.
ADVANTAGE OF THE INVENTION According to this invention, a gesture or a hand sign can be specified based on a user's finger movement or shape, and the candidate corresponding to the specified gesture or hand sign can be displayed.

Further, the present invention detects the indicator associated with the action of another user from a direction different from the detection direction of the indicator associated with the action of the user, and detects from the different direction. A storage unit that stores external dictionary information for displaying candidates corresponding to at least one of the movement, shape, and position of the indicator associated with the movement of the other user, and the display control The unit specifies the candidate corresponding to at least one of the movement, shape, and position of the indicator associated with the movement of the other user based on the external dictionary information, and determines the specified candidate Display.
According to the present invention, at least one of the movement, shape, and position of the pointer associated with the other user's motion from a direction different from the detection direction of the pointer associated with the user's motion. Can be detected. In addition, by detecting at least one of the movement, shape and position of the indicator associated with the movement of another user, the candidate corresponding to at least one of the detected movement, shape and position of the indicator is specified. Can be displayed. Therefore, in order to detect the movement, shape, and position of the pointer associated with the movement of another user, it is not necessary to detect from the detection direction of the pointer associated with the movement of the user. For this reason, it is not necessary to change the detection direction of the detection unit in order to detect the movement, shape, and position of the indicator associated with the movement of another user, and the user's effort can be reduced. it can.

The present invention further includes a communication unit that communicates with an external head-mounted display device, and the communication unit transmits the external dictionary information to the external head-mounted display device.
According to the present invention, external dictionary information can be transmitted to an external head-mounted display device that does not have external dictionary information. Accordingly, at least one of the movement, shape, and position of the indicator can be detected by an external head-mounted display device, and a candidate corresponding to the detection result can be displayed.

Further, according to the present invention, the detection unit optically detects the other user, and the display control unit is associated with the operation of the other user based on a detection result of the detection unit. In addition, a first detection area for optically detecting at least one of the movement, shape, and position of the indicator is displayed on the display area, and the display area excluding the first detection area is used for the operation of the user. A first detection area for optically detecting at least one of the movement, shape and position of the associated indicator is displayed.
According to the present invention, the first detection area for detecting the indicator associated with the user's action and the second detection area for detecting the indicator associated with the action of the other user. Is displayed. For this reason, it is not necessary to determine whether the detected indicator is an indicator corresponding to the movement of the user or an indicator corresponding to the movement of another user. For this reason, the time required for the recognition process of the indicator can be shortened, and the probability of erroneously recognizing the movement, shape and position of the indicator can be reduced.

In the present invention, the display control unit uses the candidate corresponding to at least one of the movement, shape, and position of the indicator detected by the detection unit in the first detection area as the external dictionary information. Based on the dictionary information, the candidate corresponding to at least one of the movement, shape, and position of the indicator detected by the detection unit in the second detection area is specified.
According to the present invention, a candidate corresponding to the movement, shape, or position of an indicator can be specified using a dictionary corresponding to the detection direction in which the movement, shape, and position of the indicator are detected.

  In order to solve the above problems, the present invention provides a display unit that displays an image on a display area so that the image can be seen through the outside scene and the outside scene, and at least an area that includes a range that passes through the display unit. A program that is executed by a computer mounted on a head-mounted display device mounted on the user's head, the detection unit detecting a pointer associated with an action, A procedure for displaying an image indicating a detection state in which the detection unit detects the indicator in the display area, and input content candidates corresponding to at least one of the movement, shape, and position of the indicator detected by the detection unit And a procedure for displaying.

  In order to solve the above problems, the present invention provides a display unit that displays an image on a display area so that the image can be seen through the outside scene and the outside scene, and at least an area that includes a range that passes through the display unit. A detection unit that detects a pointer associated with the movement, and a method for controlling a head-mounted display device that is mounted on the user's head, wherein the detection unit detects the pointer. Displaying an image indicating a detection state in the display area; and displaying a candidate of input content corresponding to at least one of the movement, shape, and position of the indicator detected by the detection unit. It is characterized by.

FIG. The principal part top view which shows the structure of the optical system of HMD. The perspective view which shows the structure of an image display part. The block diagram of HMD. The functional block diagram of a control apparatus. The flowchart which shows operation | movement of a control part. The figure which shows the visual field of the user who mounted | wore with HMD. The figure which shows the state by which a user's finger was displayed in the detection range image of right and left. The figure which shows the example of a display of a candidate display area. The flowchart which shows operation | movement of a control part. The figure which shows the other example of a display of a candidate display area. The figure which shows the other example of a display of a candidate display area. The figure which shows the software keyboard and character input column which were displayed on the display area. The figure which shows the display image displayed when operating the function of HMD by a hand sign or a gesture. The figure which shows the display image displayed when operating an external apparatus by a hand sign or a gesture. The figure which shows the use condition of HMD in 2nd Embodiment. The figure which shows the 1st detection range image and the 2nd detection range image which were displayed on the display area. The figure which shows the structure of the control part of 2nd Embodiment, and a memory | storage part. The figure which shows the display screen displayed on a display area.

[First Embodiment]
FIG. 1 is an external view showing an external configuration of an HMD (Head Mounted Display) 100 to which the present invention is applied.
The HMD 100 is a display device that is mounted on the user's head and includes an image display unit 20 (display unit) that allows the user to visually recognize a virtual image, and a control device 10 that controls the image display unit 20. A person wearing the image display unit 20 on the head is called a user.

  The control device 10 includes a flat box-shaped case 10A (also referred to as a housing or a main body) as shown in FIG. Case 10 </ b> A includes buttons 11, LED indicator 12, track pad 14, up / down key 15, changeover switch 16, and power switch 18. The button 11, the track pad 14, the up / down key 15, the changeover switch 16 and the power switch 18 are operated parts operated by the user. In addition, the LED indicator 12 functions as a sub display unit that indicates an operation state of the HMD 100, for example. The user can operate the HMD 100 by operating the operated part. The control device 10 functions as a controller of the HMD 100.

The image display unit 20 is a wearing body that is worn on the user's head, and has a glasses shape in the present embodiment. The image display unit 20 includes a right display unit 22, a left display unit 24, a right light guide plate 26, and a left light guide plate 28 in a main body having a right holding unit 21, a left holding unit 23, and a front frame 27.
Each of the right holding unit 21 and the left holding unit 23 extends rearward from both ends of the front frame 27, and holds the image display unit 20 on the user's head like a temple of glasses. Here, of both ends of the front frame 27, an end located on the right side of the user in the mounted state of the image display unit 20 is defined as an end ER, and an end located on the left side of the user is defined as an end EL. To do. The right holding unit 21 extends from the end ER of the front frame 27 to a position corresponding to the right side of the user when the image display unit 20 is mounted. The left holding unit 23 is provided to extend from the end EL to a position corresponding to the left side of the user when the image display unit 20 is mounted.

  The right light guide plate 26 and the left light guide plate 28 are provided on the front frame 27. The right light guide plate 26 is positioned in front of the right eye of the user when the image display unit 20 is mounted, and causes the right eye to visually recognize the image. The left light guide plate 28 is positioned in front of the left eye of the user when the image display unit 20 is mounted, and causes the left eye to visually recognize the image.

  The front frame 27 has a shape in which one end of the right light guide plate 26 and one end of the left light guide plate 28 are connected to each other, and this connection position is in a wearing state in which the user wears the image display unit 20. Corresponding to the eyebrows. The front frame 27 may be provided with a nose pad portion that comes into contact with the user's nose when the image display unit 20 is mounted at a connection position between the right light guide plate 26 and the left light guide plate 28. In this case, the image display unit 20 can be held on the user's head by the nose pad, the right holding unit 21 and the left holding unit 23. Further, a belt (not shown) that contacts the back of the user when the image display unit 20 is mounted may be connected to the right holding unit 21 and the left holding unit 23. In this case, the image display unit 20 is used by the belt. Can be held on the person's head.

  The right display unit 22 realizes image display by the right light guide plate 26. The right display unit 22 is provided in the right holding unit 21 and is located in the vicinity of the right side of the user in the mounted state. The left display unit 24 realizes image display by the left light guide plate 28. The left display unit 24 is provided in the left holding part 23 and is located in the vicinity of the left side of the user in the wearing state.

The right light guide plate 26 and the left light guide plate 28 of the present embodiment are optical units formed of a light-transmitting resin or the like, and are, for example, prisms, and output image light output from the right display unit 22 and the left display unit 24. Lead to the eyes of the user.
A light control plate (not shown) may be provided on the surfaces of the right light guide plate 26 and the left light guide plate 28. The light control plate is an optical element on a thin plate having different transmittance depending on the wavelength region of light, and functions as a so-called wavelength filter. For example, the light control plate is disposed so as to cover the front side of the front frame 27 which is the side opposite to the eye side of the user. By appropriately selecting the optical characteristics of the light control plate, the transmittance of light in an arbitrary wavelength region such as visible light, infrared light, and ultraviolet light can be adjusted. The amount of external light incident on the light plate 28 and transmitted through the right light guide plate 26 and the left light guide plate 28 can be adjusted.

  The image display unit 20 guides the image light generated by the right display unit 22 and the left display unit 24 to the right light guide plate 26 and the left light guide plate 28, respectively. The image light guided to the right light guide plate 26 and the left light guide plate 28 enters the right eye and the left eye of the user and makes the user visually recognize a virtual image. Thereby, the image display unit 20 displays an image.

  When the external light enters the user's eyes through the right light guide plate 26 and the left light guide plate 28 from the front of the user, the image light and the external light constituting the virtual image enter the user's eyes. In other words, the visibility of the virtual image is affected by the intensity of external light. For this reason, for example, by attaching a light control plate to the front frame 27 and appropriately selecting or adjusting the optical characteristics of the light control plate, the ease of visual recognition of the virtual image can be adjusted. In a typical example, it is possible to use a light control plate having a light transmittance that allows a user wearing the HMD 100 to visually recognize at least the outside scenery. In addition, when the dimming plate is used, it is possible to protect the right light guide plate 26 and the left light guide plate 28 and to suppress the damage to the right light guide plate 26 and the left light guide plate 28, the adhesion of dirt, and the like. The light control plate may be detachable with respect to the front frame 27 or each of the right light guide plate 26 and the left light guide plate 28, or may be mounted by replacing a plurality of types of light control plates. May be omitted.

  The camera 61 is disposed on the front frame 27 of the image display unit 20. The configuration and arrangement of the camera 61 are determined so as to capture an outside scene direction visually recognized by the user while wearing the image display unit 20. For example, the camera 61 is provided on the front surface of the front frame 27 at a position that does not block external light that passes through the right light guide plate 26 and the left light guide plate 28. In the example of FIG. 1, the camera 61 is disposed on the end ER side of the front frame 27, but the camera 61 may be disposed on the end EL side, and the right light guide plate 26 and the left light guide plate 28. You may arrange | position in the connection part. The camera 61 corresponds to the “detection unit” of the present invention. Further, the range of the outside scene that is visually recognized by the user wearing the image display unit 20 as the imaging range of the camera 61 corresponds to the “region including at least the range that transmits the display unit” of the present invention. The camera 61 optically detects the indicator or the user.

The camera 61 is a digital camera that includes an imaging element such as a CCD or CMOS, an imaging lens, and the like. The camera 61 of this embodiment is a monocular camera, but may be a stereo camera. The camera 61 images at least a part of the outside scene (real space) in the front side direction of the HMD 100, in other words, the user's viewing direction when the HMD 100 is worn. In another expression, the camera 61 captures a range or direction that overlaps the user's field of view, and captures a direction in which the user gazes. The direction and width of the angle of view of the camera 61 can be set as appropriate. In the present embodiment, as will be described later, the angle of view of the camera 61 includes the outside world visually recognized by the user through the right light guide plate 26 and the left light guide plate 28. More preferably, the angle of view of the camera 61 is set so that the entire field of view of the user who can see through the right light guide plate 26 and the left light guide plate 28 can be imaged.
The camera 61 performs imaging according to the control of the imaging control unit 147 included in the control unit 150 (FIG. 5). The camera 61 outputs captured image data to the control unit 150 via an interface 211 described later.

  The HMD 100 may include a distance sensor (not shown) that detects a distance to a measurement object positioned in a preset measurement direction. The distance sensor can be disposed, for example, at a connection portion between the right light guide plate 26 and the left light guide plate 28 in the front frame 27. In this case, when the image display unit 20 is mounted, the position of the distance sensor is substantially in the middle between the user's eyes in the horizontal direction and above the user's eyes in the vertical direction. The measurement direction of the distance sensor can be, for example, the front side direction of the front frame 27, in other words, the direction overlapping the imaging direction of the camera 61. The distance sensor can be configured to include, for example, a light source such as an LED or a laser diode, and a light receiving unit that receives reflected light that is reflected from the light to be measured. The distance sensor may perform a triangulation process or a distance measurement process based on a time difference in accordance with the control of the control unit 150. The distance sensor may be configured to include a sound source that emits ultrasonic waves and a detection unit that receives ultrasonic waves reflected by the measurement object. In this case, the distance sensor may perform the distance measurement process based on the time difference until the reflection of the ultrasonic wave according to the control of the control unit 150.

FIG. 2 is a principal plan view showing the configuration of the optical system provided in the image display unit 20. FIG. 2 shows the user's left eye LE and right eye RE for explanation.
As shown in FIG. 2, the right display unit 22 and the left display unit 24 are configured symmetrically. The right display unit 22 includes an OLED (Organic Light Emitting Diode) unit 221 that emits image light and a lens group that guides the image light L emitted from the OLED unit 221 as a configuration that allows the user's right eye RE to visually recognize an image. Right optical system 251. The image light L is guided to the right light guide plate 26 by the right optical system 251.

The OLED unit 221 includes an OLED panel 223 and an OLED drive circuit 225 that drives the OLED panel 223. The OLED panel 223 is a self-luminous display panel configured by arranging, in a matrix, light emitting elements that emit light of R (red), G (green), and B (blue) by emitting light by organic electroluminescence. It is. The OLED panel 223 includes a plurality of pixels, each of which includes one R, G, and B element, and forms an image with pixels arranged in a matrix. The OLED drive circuit 225 selects a light emitting element included in the OLED panel 223 and energizes the light emitting element under the control of the control unit 150 (FIG. 5), and causes the light emitting element of the OLED panel 223 to emit light. The OLED drive circuit 225 is fixed to the back surface of the OLED panel 223, that is, the back surface of the light emitting surface by bonding or the like. The OLED drive circuit 225 may be formed of a semiconductor device that drives the OLED panel 223, for example, and may be mounted on a substrate (not shown) fixed to the back surface of the OLED panel 223. A temperature sensor 217 is mounted on this substrate.
Note that the OLED panel 223 may have a configuration in which light emitting elements that emit white light are arranged in a matrix, and color filters corresponding to R, G, and B colors are stacked. Further, an OLED panel 223 having a WRGB configuration provided with a light emitting element that emits W (white) light in addition to the light emitting elements that respectively emit R, G, and B color light may be used.

  The right optical system 251 includes a collimator lens that converts the image light L emitted from the OLED panel 223 into a parallel light flux. The image light L converted into a parallel light beam by the collimator lens enters the right light guide plate 26. A plurality of reflecting surfaces that reflect the image light L are formed in an optical path that guides light inside the right light guide plate 26. The image light L is guided to the right eye RE side through a plurality of reflections inside the right light guide plate 26. A half mirror 261 (reflection surface) located in front of the right eye RE is formed on the right light guide plate 26. The image light L is reflected by the half mirror 261 and emitted from the right light guide plate 26 toward the right eye RE. The image light L forms an image on the retina of the right eye RE, and allows the user to visually recognize the image.

  In addition, the left display unit 24 has a left optical system including an OLED unit 241 that emits image light and a lens group that guides the image light L emitted from the OLED unit 241 as a configuration that allows the user's left eye LE to visually recognize an image. 252. The image light L is guided to the left light guide plate 28 by the left optical system 252.

  The OLED unit 241 includes an OLED panel 243 and an OLED drive circuit 245 that drives the OLED panel 243. The OLED panel 243 is a self-luminous display panel configured similarly to the OLED panel 223. The OLED driving circuit 245 causes the light emitting elements of the OLED panel 243 to emit light by selecting a light emitting element included in the OLED panel 243 and energizing the light emitting element according to the control of the control unit 150 (FIG. 5). The OLED drive circuit 245 is fixed to the back surface of the OLED panel 243, that is, the back surface of the light emitting surface by bonding or the like. The OLED drive circuit 245 may be configured by a semiconductor device that drives the OLED panel 243, for example, and may be mounted on a substrate (not shown) fixed to the back surface of the OLED panel 243. A temperature sensor 239 is mounted on this substrate.

  The left optical system 252 includes a collimating lens that converts the image light L emitted from the OLED panel 243 into a light beam in a parallel state. The image light L converted into a parallel light beam by the collimator lens enters the left light guide plate 28. The left light guide plate 28 is an optical element formed with a plurality of reflecting surfaces that reflect the image light L, and is, for example, a prism. The image light L is guided to the left eye LE side through a plurality of reflections inside the left light guide plate 28. The left light guide plate 28 is formed with a half mirror 281 (reflection surface) located in front of the left eye LE. The image light L is reflected by the half mirror 281 and emitted from the left light guide plate 28 toward the left eye LE. The image light L forms an image on the retina of the left eye LE, and allows the user to visually recognize the image.

According to this configuration, the HMD 100 functions as a see-through display device. That is, the image light L reflected by the half mirror 261 and the external light OL transmitted through the right light guide plate 26 are incident on the user's right eye RE. Further, the image light L reflected by the half mirror 281 and the external light OL transmitted through the half mirror 281 are incident on the left eye LE. In this way, the HMD 100 superimposes the image light L of the internally processed image and the external light OL so as to enter the user's eyes, and for the user, the outside light is transmitted through the right light guide plate 26 and the left light guide plate 28. The image by the image light L is visually recognized over the outside scene.
The half mirrors 261 and 281 are image extraction units that extract image by reflecting image light output from the right display unit 22 and the left display unit 24, respectively, and can be called display units.

  The left optical system 252 and the left light guide plate 28 are collectively referred to as a “left light guide”, and the right optical system 251 and the right light guide plate 26 are collectively referred to as a “right light guide”. The configuration of the right light guide and the left light guide is not limited to the above example, and any method can be used as long as a virtual image is formed in front of the user's eyes using image light. It may be used, or a semi-transmissive reflective film may be used.

Returning to FIG. 1, the control device 10 and the image display unit 20 are connected by a connection cable 40. The connection cable 40 is detachably connected to a connector (not shown) provided at the lower part of the case 10 </ b> A, and is connected to various circuits provided in the image display unit 20 from the tip of the left holding unit 23. The connection cable 40 includes a metal cable that transmits digital data or an optical fiber cable, and may include a metal cable that transmits an analog signal. A connector 46 is provided in the middle of the connection cable 40. The connector 46 is a jack for connecting a stereo mini plug, and the connector 46 and the control device 10 are connected by a line for transmitting an analog audio signal, for example. In the configuration example shown in FIG. 1, a headset 30 having a right earphone 32 and a left earphone 34 and a microphone 63 constituting a stereo headphone is connected to a connector 46.
The control device 10 and the image display unit 20 may be wirelessly connected. For example, the control device 10 and the image display unit 20 mutually transmit and receive control signals and data by wireless communication complying with standards such as Bluetooth (registered trademark) and wireless LAN (including Wi-Fi (registered trademark)). It is good also as a structure.

  For example, as shown in FIG. 1, the microphone 63 is arranged so that the sound collection unit of the microphone 63 faces the user's line of sight, collects sound, and outputs a sound signal to the sound interface 182 (FIG. 4). To do. For example, the microphone 63 may be a monaural microphone or a stereo microphone, may be a directional microphone, or may be an omnidirectional microphone. The microphone 63 corresponds to the “voice input unit” of the present invention.

  The control device 10 includes a button 11, an LED indicator 12, a track pad 14, an up / down key 15, a changeover switch 16, and a power switch 18 as operated parts operated by a user. These operated parts are arranged on the surface of the case 10A.

The button 11 includes keys and switches for operating the control device 10, and these keys and switches are displaced by a pressing operation. For example, the button 11 includes a menu key, a home key, and a “return” key for performing operations related to the operating system 143 (see FIG. 5) executed by the control device 10.
The LED indicator 12 lights up or blinks in accordance with the operating state of the HMD 100. The up / down key 15 is used to input an instruction to increase / decrease the volume output from the right earphone 32 and the left earphone 34 and to input an instruction to increase / decrease the display brightness of the image display unit 20. The changeover switch 16 is a switch for changing an input corresponding to the operation of the up / down key 15. The power switch 18 is a switch that switches on / off the power of the HMD 100, and is, for example, a slide switch.

  The track pad 14 has an operation surface for detecting a contact operation, and outputs an operation signal according to an operation on the operation surface. The detection method on the operation surface is not limited, and an electrostatic method, a pressure detection method, an optical method, or other methods can be adopted. Contact (touch operation) to the track pad 14 is detected by a touch sensor (not shown). An LED display unit 17 is installed on the track pad 14. The LED display unit 17 includes a plurality of LEDs, and light of each LED is transmitted through the track pad 14 to display an operation icon or the like. This icon or the like functions as a software button.

FIG. 3 is a perspective view showing a configuration of the image display unit 20 and shows a main configuration of the image display unit 20 as viewed from the user's head side. FIG. 3 shows the side of the image display unit 20 that contacts the user's head, in other words, the side that can be seen by the user's right eye RE and left eye LE. In other words, the back sides of the right light guide plate 26 and the left light guide plate 28 are visible.
In FIG. 3, the half mirror 261 that irradiates the user's right eye RE with the image light and the half mirror 281 that irradiates the left eye LE with the image light appear as substantially rectangular areas. Further, the entire right light guide plate 26 and left light guide plate 28 including the half mirrors 261 and 281 transmit external light as described above. For this reason, the user sees the outside scene through the entire right light guide plate 26 and the left light guide plate 28, and sees a rectangular display image at the positions of the half mirrors 261 and 281.

  The camera 61 is disposed at the right end of the image display unit 20 and images the direction in which both eyes of the user face, that is, the front for the user. The optical axis of the camera 61 is a direction including the line-of-sight directions of the right eye RE and the left eye LE. The outside scene that the user can visually recognize in a state where the HMD 100 is worn is not always at infinity. For example, when a user looks at an object located in front of the user with both eyes, the distance from the user to the object is often about 30 cm to 10 m, and is about 1 m to 4 m. But more. Therefore, for the HMD 100, an upper limit and a lower limit of the distance from the user to the object during normal use may be determined. This standard may be obtained by investigation or experiment, or may be set by the user. The optical axis and the angle of view of the camera 61 are set to the angle of view when the distance to the object during normal use corresponds to the set upper limit guide and the lower limit guide. It is preferable to be set so as to be included.

  In general, the viewing angle of a human is about 200 degrees in the horizontal direction and about 125 degrees in the vertical direction. Among them, the effective field of view with excellent information receiving ability is about 30 degrees in the horizontal direction and about 20 degrees in the vertical direction. Furthermore, the stable gaze field in which the gaze point that the human gazes at appears to be quickly and stably is about 60 to 90 degrees in the horizontal direction and about 45 to 70 degrees in the vertical direction. When the gazing point is an object located in front of the user, in the user's field of view, about 30 degrees in the horizontal direction and about 20 degrees in the vertical direction are effective around the line of sight of the right eye RE and the left eye LE. It is a field of view. Further, the stable viewing field is about 60 to 90 degrees in the horizontal direction and about 45 to 70 degrees in the vertical direction, and the viewing angle is about 200 degrees in the horizontal direction and about 125 degrees in the vertical direction. Furthermore, the actual visual field that the user sees through the right light guide plate 26 and the left light guide plate 28 can be referred to as a real field of view (FOV: Field Of View). In the configuration of the present embodiment shown in FIGS. 1 and 2, the actual visual field corresponds to an actual visual field that is transmitted through the right light guide plate 26 and the left light guide plate 28 and visually recognized by the user. The real field of view is narrower than the viewing angle and stable field of view, but wider than the effective field of view.

  The field angle of the camera 61 is preferably capable of capturing an image in a wider range than the user's field of view, and specifically, the field angle is preferably at least wider than the user's effective field of view. Further, it is more preferable that the angle of view is wider than the actual field of view of the user. More preferably, the angle of view is wider than the stable viewing field of the user, and most preferably, the angle of view is wider than the viewing angle of both eyes of the user.

  The camera 61 may include a so-called wide-angle lens as an imaging lens so that a wide angle of view can be captured. The wide-angle lens may include a lens called a super-wide-angle lens or a quasi-wide-angle lens, may be a single focus lens or a zoom lens, and the camera 61 includes a lens group including a plurality of lenses. There may be.

FIG. 4 is a block diagram illustrating a configuration of each unit included in the HMD 100.
The control device 10 includes a main processor 140 that executes a program and controls the HMD 100. A memory 118 and a non-volatile storage unit 121 are connected to the main processor 140. The main processor 140 is connected to the track pad 14 and the operation unit 110 as input devices. In addition, a 6-axis sensor 111 and a magnetic sensor 113 are connected to the main processor 140 as sensors. The main processor 140 is connected to a GPS receiver 115, a communication unit 117, a beacon receiver 119, an audio codec 180, an external connector 184, an external memory interface 186, a USB connector 188, a sensor hub 192, and an FPGA 194. These function as an interface with the outside. The communication unit 117 corresponds to the “control information output unit” of the present invention.

  The main processor 140 is mounted on the controller board 120 built in the control device 10. In addition to the main processor 140, a memory 118, a nonvolatile storage unit 121, and the like may be mounted on the controller board 120. In the present embodiment, a six-axis sensor 111, a magnetic sensor 113, a GPS receiver 115, a communication unit 117, a memory 118, a nonvolatile storage unit 121, an audio codec 180, and the like are mounted on the controller board 120. Further, the external connector 184, the external memory interface 186, the USB connector 188, the sensor hub 192, the FPGA 194, and the interface 196 may be mounted on the controller board 120.

  The memory 118 constitutes a work area that temporarily stores a program to be executed and data to be processed when the main processor 140 executes the program. The non-volatile storage unit 121 is configured by a flash memory or an eMMC (embedded Multi Media Card). The nonvolatile storage unit 121 stores a program executed by the main processor 140 and various data processed by the main processor 140 executing the program.

The main processor 140 detects a touch operation on the operation surface of the track pad 14 based on an operation signal input from the track pad 14 and acquires an operation position.
The operation unit 110 includes a button 11 and an LED display unit 17. The operation unit 110 outputs an operation signal corresponding to the operated operation element to the main processor 140 when an operation element such as a button or a switch included in the button 11 is operated.

  The LED display unit 17 controls the turning on and off of the LED indicator 12 according to the control of the main processor 140. The LED display unit 17 may include an LED (not shown) disposed immediately below the track pad 14 and a drive circuit that lights the LED. In this case, the LED display unit 17 turns on, blinks, and turns off the LED according to the control of the main processor 140.

The 6-axis sensor 111 is a motion sensor (inertial sensor) including a 3-axis acceleration sensor and a 3-axis gyro (angular velocity) sensor. The 6-axis sensor 111 may employ an IMU (Internal Measurement Unit) in which the above sensors are modularized.
The magnetic sensor 113 is, for example, a triaxial geomagnetic sensor.
The 6-axis sensor 111 and the magnetic sensor 113 output the detection value to the main processor 140 according to a sampling period specified in advance. The 6-axis sensor 111 and the magnetic sensor 113 output detection values to the main processor 140 at a timing designated by the main processor 140 in response to a request from the main processor 140.

  The GPS receiving unit 115 includes a GPS antenna (not shown) and receives a GPS signal transmitted from a GPS satellite. The GPS receiving unit 115 outputs the received GPS signal to the main processor 140. Further, the GPS receiving unit 115 measures the signal strength of the received GPS signal and outputs it to the main processor 140. Information such as received signal strength (RSSI: Received Signal Strength Indication), electric field strength, magnetic field strength, signal to noise ratio (SNR) can be used as the signal strength.

  The communication unit 117 performs wireless communication with an external device. The communication unit 117 includes an antenna, an RF circuit, a baseband circuit, a communication control circuit, and the like, or a device in which these are integrated. The communication unit 117 performs wireless communication complying with standards such as Bluetooth and wireless LAN (including Wi-Fi), for example.

  The audio interface 182 is an interface for inputting and outputting audio signals. In the present embodiment, the audio interface 182 includes a connector 46 (FIG. 1) provided on the connection cable 40. The connector 46 is connected to the headset 30. The audio signal output from the audio interface 182 is input to the right earphone 32 and the left earphone 34, whereby the right earphone 32 and the left earphone 34 output audio. Further, the microphone 63 provided in the headset 30 collects sound and outputs a sound signal to the sound interface 182. An audio signal input from the microphone 63 to the audio interface 182 is input to the external connector 184.

  The audio codec 180 is connected to the audio interface 182 and encodes and decodes an audio signal input / output via the audio interface 182. The audio codec 180 may include an A / D converter that converts analog audio signals to digital audio data, and a D / A converter that performs the reverse conversion. For example, the HMD 100 according to the present embodiment outputs sound by the right earphone 32 and the left earphone 34 and collects sound by the microphone 63. The audio codec 180 converts the digital audio data output from the main processor 140 into an analog audio signal and outputs the analog audio signal via the audio interface 182. The audio codec 180 converts an analog audio signal input to the audio interface 182 into digital audio data and outputs the digital audio data to the main processor 140.

  The external connector 184 is a connector that connects an external device that communicates with the main processor 140. The external connector 184 is an interface for connecting an external device when, for example, an external device is connected to the main processor 140 to debug a program executed by the main processor 140 or collect a log of the operation of the HMD 100. It is.

The external memory interface 186 is an interface to which a portable memory device can be connected. For example, the external memory interface 186 includes a memory card slot in which a card type recording medium is mounted and data can be read and an interface circuit. In this case, the size, shape and standard of the card type recording medium are not limited and can be changed as appropriate.
The USB (Universal Serial Bus) connector 188 includes a connector conforming to the USB standard and an interface circuit. The USB connector 188 can connect a USB memory device, a smartphone, a computer, or the like. The size and shape of the USB connector 188 and the compatible USB standard version can be appropriately selected and changed.

  The sensor hub 192 and the FPGA 194 are connected to the image display unit 20 via an interface (I / F) 196. The sensor hub 192 acquires detection values of various sensors included in the image display unit 20 and outputs them to the main processor 140. The FPGA 194 executes processing of data transmitted / received between the main processor 140 and each unit of the image display unit 20 and transmission via the interface 196.

  The HMD 100 includes a vibrator 19. The vibrator 19 includes a motor and an eccentric rotor (both not shown), and may have other necessary configurations. The vibrator 19 generates vibrations by rotating the motor according to the control of the main processor 140. For example, the HMD 100 generates vibration by the vibrator 19 with a predetermined vibration pattern when an operation on the operation unit 110 is detected, when the power of the HMD 100 is turned on / off, or in other cases.

  The right display unit 22 and the left display unit 24 of the image display unit 20 are each connected to the control device 10. As shown in FIG. 1, in the HMD 100, a connection cable 40 is connected to the left holding unit 23, wiring connected to the connection cable 40 is laid inside the image display unit 20, and each of the right display unit 22 and the left display unit 24 is Connected to the control device 10.

The right display unit 22 includes a display unit substrate 210. On the display unit substrate 210, an interface (I / F) 211 connected to the interface 196, a receiving unit (Rx) 213 that receives data input from the control device 10 via the interface 211, and an EEPROM 215 are mounted. .
The interface 211 connects the receiving unit 213, the EEPROM 215, the temperature sensor 217, the camera 61, the illuminance sensor 65, and the LED indicator 67 to the control device 10.

  An EEPROM (Electrically Erasable Programmable Read-Only Memory) 215 stores various data so that the main processor 140 can read the data. The EEPROM 215 stores, for example, data relating to light emission characteristics and display characteristics of the OLED units 221 and 241 provided in the image display unit 20, data relating to characteristics of sensors provided in the right display unit 22 or the left display unit 24, and the like. Specifically, parameters relating to gamma correction of the OLED units 221, 241 and data for compensating the detection values of the temperature sensors 217, 239 are stored. These data are generated by the factory inspection of the HMD 100 and written in the EEPROM 215. After the shipment, the main processor 140 can perform processing using the data in the EEPROM 215.

The camera 61 executes imaging according to a signal input via the interface 211 and outputs captured image data or a signal indicating the imaging result to the control device 10.
As shown in FIG. 1, the illuminance sensor 65 is provided at an end ER of the front frame 27 and is arranged to receive external light from the front of the user wearing the image display unit 20. The illuminance sensor 65 outputs a detection value corresponding to the amount of received light (received light intensity).
As shown in FIG. 1, the LED indicator 67 is disposed near the camera 61 at the end ER of the front frame 27. The LED indicator 67 is lit during execution of imaging by the camera 61 to notify that imaging is in progress.

  The temperature sensor 217 detects the temperature and outputs a voltage value or a resistance value corresponding to the detected temperature as a detected value. The temperature sensor 217 is mounted on the back side of the OLED panel 223 (FIG. 2). For example, the temperature sensor 217 may be mounted on the same substrate as the OLED drive circuit 225. With this configuration, the temperature sensor 217 mainly detects the temperature of the OLED panel 223.

  The receiving unit 213 receives data transmitted from the main processor 140 via the interface 211. When receiving the image data of the image displayed on the OLED unit 221, the receiving unit 213 outputs the received image data to the OLED drive circuit 225 (FIG. 2).

The left display unit 24 includes a display unit substrate 210. On the display unit substrate 210, an interface (I / F) 231 connected to the interface 196 and a receiving unit (Rx) 233 that receives data input from the control device 10 via the interface 231 are mounted. In addition, a six-axis sensor 235 and a magnetic sensor 237 are mounted on the display unit substrate 210.
The interface 231 connects the receiving unit 233, the six-axis sensor 235, the magnetic sensor 237, and the temperature sensor 239 to the control device 10.

The 6-axis sensor 235 is a motion sensor (inertial sensor) including a 3-axis acceleration sensor and a 3-axis gyro (angular velocity) sensor. The 6-axis sensor 235 may employ an IMU in which the above sensors are modularized.
The magnetic sensor 237 is, for example, a triaxial geomagnetic sensor.

The temperature sensor 239 detects the temperature and outputs a voltage value or a resistance value corresponding to the detected temperature as a detected value. The temperature sensor 239 is mounted on the back side of the OLED panel 243 (FIG. 2). The temperature sensor 239 may be mounted on the same substrate as the OLED drive circuit 245, for example. With this configuration, the temperature sensor 239 mainly detects the temperature of the OLED panel 243.
Further, the temperature sensor 239 may be incorporated in the OLED panel 243 or the OLED drive circuit 245. The substrate may be a semiconductor substrate. Specifically, when the OLED panel 243 is mounted as an Si-OLED as an integrated circuit on an integrated semiconductor chip together with the OLED drive circuit 245 and the like, the temperature sensor 239 may be mounted on the semiconductor chip.

The camera 61, the illuminance sensor 65, the temperature sensor 217, and the 6-axis sensor 235, the magnetic sensor 237, and the temperature sensor 239 included in the left display unit 24 are connected to the sensor hub 192. The sensor hub 192 sets and initializes the sampling period of each sensor according to the control of the main processor 140. The sensor hub 192 executes energization to each sensor, transmission of control data, acquisition of a detection value, and the like in accordance with the sampling period of each sensor. The sensor hub 192 outputs detection values of the sensors included in the right display unit 22 and the left display unit 24 to the main processor 140 at a preset timing. The sensor hub 192 may have a function of temporarily holding the detection value of each sensor in accordance with the output timing to the main processor 140. In addition, the sensor hub 192 may have a function of converting the data of the output value of each sensor or the data format into data of a unified data format and outputting the data to the main processor 140.
The sensor hub 192 starts and stops energization of the LED indicator 67 according to the control of the main processor 140, and lights or blinks the LED indicator 67 in accordance with the timing when the camera 61 starts and ends imaging.

  The control device 10 includes a power supply unit 130 and operates with power supplied from the power supply unit 130. The power supply unit 130 includes a rechargeable battery 132 and a power supply control circuit 134 that detects the remaining capacity of the battery 132 and controls charging of the battery 132. The power control circuit 134 is connected to the main processor 140 and outputs a detected value of the remaining capacity of the battery 132 or a detected value of the voltage to the main processor 140. Further, the power may be supplied from the control device 10 to the image display unit 20 based on the power supplied from the power supply unit 130. The power supply state from the power supply unit 130 to each unit of the control device 10 and the image display unit 20 may be configured to be controllable by the main processor 140.

  FIG. 5 is a functional block diagram of the storage unit 122 and the control unit 150 that constitute the control system of the control device 10. The storage unit 122 illustrated in FIG. 5 is a logical storage unit configured by the nonvolatile storage unit 121 (FIG. 4), and may include an EEPROM 215. The control unit 150 and various functional units included in the control unit 150 are formed by cooperation of software and hardware by the main processor 140 executing a program. The control unit 150 and each functional unit constituting the control unit 150 include, for example, a main processor 140, a memory 118, and a nonvolatile storage unit 121.

The control unit 150 executes various processes using data stored in the storage unit 122 and controls the HMD 100.
The storage unit 122 stores various data processed by the control unit 150. Specifically, the storage unit 122 stores setting data 123, content data 124, a pattern registration dictionary 125, and an audio pattern dictionary 126.

  The setting data 123 includes various setting values that set the operation of the HMD 100. Further, when the control unit 150 controls the HMD 100, parameters, determinants, arithmetic expressions, LUTs (Look Up Tables), and the like may be included in the setting data 123.

The content data 124 is content data including a display image and video displayed by the image display unit 20 under the control of the control unit 150, and includes image data or video data. Further, the content data 124 may include audio data. The content data 124 may include image data of a plurality of images. In this case, the plurality of images are not limited to images displayed on the image display unit 20 at the same time.
The content data 124 is an interactive type in which when the content is displayed on the image display unit 20, the control device 10 receives a user operation and the control unit 150 executes a process according to the received operation. It may be content. In this case, the content data 124 may include image data of a menu screen that is displayed when an operation is accepted, data that defines processing corresponding to items included in the menu screen, and the like.

The pattern registration dictionary 125 includes a gesture registration dictionary 1251 and a hand sign registration dictionary 1252.
The gesture registration dictionary 1251 is a dictionary in which information specifying a gesture and information indicating characters and symbols (hereinafter referred to as character information) are registered in association with each other. Here, the gesture is a movement of a finger that performs a preset finger shape in a preset order. The information for specifying the gesture is information indicating the shape and positional relationship of the fingers for each state of the fingers constituting the gesture. For example, it is assumed that the gesture is an operation of opening a grasped hand, that is, an operation of opening a hand in a “goo” state to a “par” state. In this case, the information specifying the gesture includes information indicating the shape and positional relationship of the finger that specifies the “go” state, and information indicating the shape and positional relationship of the finger that specifies the “par” state. It is.

  The hand sign registration dictionary 1252 is a dictionary in which information for specifying a hand sign and character information are associated and registered. The hand sign indicates a preset static shape or position of the hand or finger, and the information for specifying the hand sign is information indicating the position or shape of the hand or finger.

Further, information for specifying a gesture and functions of the HMD 100 or an external device operated by the specified gesture may be registered in the gesture registration dictionary 1251 in association with each other. Similarly, information for specifying a hand sign and functions of the HMD 100 or an external device operated by the specified hand sign may be registered in the hand sign registration dictionary 1252 in association with each other.
For example, when the audio function of the HMD 100 is operated by a hand sign or gesture, the operation items of the audio function such as volume control and music selection are registered in association with information for specifying the hand sign and gesture associated with the operation item. To do.
When an external device is operated by a hand sign or gesture, an operation item capable of operating the external device is registered in association with information for specifying a hand sign or gesture associated with the operation item. For example, when the external device is an electric knife, the operation items for turning on / off the electric knife and for increasing / decreasing the amount of current flowing to the electric knife are associated with the operation items. Register in association with information identifying hand signs and gestures.

When different operations are performed for the left hand and the right hand, different operations are associated with the left hand hand sign and gesture and the right hand hand sign and gesture, respectively.
That is, in the gesture registration dictionary 1251, information specifying the left hand gesture and an operation associated with the gesture are registered in association with each other. In the gesture registration dictionary 1251, information for specifying the right hand gesture and an operation associated with the gesture are registered in association with each other. In the hand sign registration dictionary 1252, information for specifying the hand sign of the left hand and an operation associated with the hand sign are registered in association with each other. Also, in the hand sign registration dictionary 1252, information for specifying the hand sign of the right hand and an operation associated with the hand sign are registered in association with each other.

In addition, the storage unit 122 stores an audio pattern dictionary 126.
The voice pattern dictionary 126 is a dictionary in which preset voice data is registered. In the voice pattern dictionary 126, for example, finger snap voice data, which is an operation of ringing a finger, or knock voice data for hitting the track pad 14 or the like is registered.

  The control unit 150 has functions of an operating system (OS) 143, an image processing unit 145, an imaging control unit 147, an operation control unit 149, a detection control unit 151, and a display control unit 153. The imaging control unit 147 corresponds to the “detection unit” of the present invention. The detection control unit 151 corresponds to the “operation detection unit” of the present invention.

  The functions of the operating system 143 are functions of a control program stored in the storage unit 122, and the other units are functions of application programs executed on the operating system 143.

The image processing unit 145 generates a signal to be transmitted to the right display unit 22 and the left display unit 24 based on image data of an image or video displayed by the image display unit 20. The signal generated by the image processing unit 145 may be a vertical synchronization signal, a horizontal synchronization signal, a clock signal, an analog image signal, or the like.
Further, the image processing unit 145 may perform resolution conversion processing for converting the resolution of the image data to a resolution suitable for the right display unit 22 and the left display unit 24 as necessary. The image processing unit 145 also performs image adjustment processing for adjusting the brightness and saturation of image data, 2D image data from 3D image data, 2D / 3D conversion processing for generating 3D image data from 2D image data, and the like. May be executed. When these image processes are executed, the image processing unit 145 generates a signal for displaying an image based on the processed image data, and transmits the signal to the image display unit 20 via the connection cable 40.

  The image processing unit 145 may be configured by hardware different from the main processor 140 (for example, a digital signal processor (DSP)), in addition to the configuration realized by the main processor 140 executing a program.

  The imaging control unit 147 controls the camera 61 to execute imaging, generates captured image data, and temporarily stores the captured image data in the storage unit 122. When the camera 61 is configured as a camera unit including a circuit that generates captured image data, the imaging control unit 147 acquires captured image data from the camera 61 and temporarily stores the captured image data in the storage unit 122.

The operation control unit 149 detects an operation on the track pad 14 and the operation unit 110 and outputs data corresponding to the detected operation. For example, when a button or the like of the operation unit 110 is operated, the operation control unit 149 generates operation data indicating the operation content and outputs the operation data to the display control unit 153. The display control unit 153 changes the display state of the image display unit 20 according to the operation data input from the operation control unit 149.
Further, the operation control unit 149 acquires the coordinates of the operation position on the track pad 14 when the operation on the track pad 14 is detected. The operation control unit 149 generates a locus of the operation position. The locus of the operation position is the locus of the operation position while the touch operation on the track pad 14 is not released, that is, while the state where the operation body is in contact with the track pad 14 continues.

  The detection control unit 151 detects an indicator associated with the user's action (movement) from the captured image data of the camera 61. As the indicator, for example, a hand or a finger including a user's arm (hereinafter, the hand and the finger are collectively referred to as a finger) can be used. Moreover, you may use the member with which the user's finger | toe and arm were mounted | worn as an indicator. For example, a ring attached to the user's finger, a watch or a band attached to the user's arm, and the like can be used as the indicator. Since these indicators are things that the user wears, if the user moves the fingers or arms, the indicators move together (together) according to the movement of the fingers or arms. Moreover, it is also possible to use a preset tool such as a surgical tool (for example, an electric knife) held by the user as the indicator. The user's left hand corresponds to a “first indicator”, and the user's right hand corresponds to a “second indicator”.

The detection control unit 151 acquires captured image data from the storage unit 122, and detects a user's hand or finger region (hereinafter referred to as a finger region) captured in the acquired captured image data. In addition, the detection control unit 151 identifies the shape and position of the finger reflected in the detected finger region, compares the identified shape and position of the finger with information registered in the pattern registration dictionary 125, and determines the gesture or hand. Detect signatures.
Specifically, the detection control unit 151 detects a region where the user's finger is imaged from the acquired captured image data. For example, the detection control unit 151 extracts a skin color region from the captured image data region by skin color extraction processing, and detects a finger region by comparing the pixel value of the extracted skin color region with a threshold value by threshold processing. To do. In addition, the detection control unit 151 extracts the contour of the obtained finger region by edge detection processing, and detects the shape and position of the finger using feature point matching or curvature determination based on the contour unevenness. Further, the detection control unit 151 detects the shape and position of the finger from each of the captured image data that is continuously captured, and detects a change in the shape and position of the finger, that is, a movement. When detecting the shape and position of the finger, the detection control unit 151 compares the detected shape and position of the finger with information registered in the pattern registration dictionary 125 to detect a gesture and a hand sign.

  The display control unit 153 generates a control signal for controlling the right display unit 22 and the left display unit 24, and controls the right display unit 22 and the left display unit 24 based on the generated control signal. Specifically, the display control unit 151 controls the OLED drive circuits 225 and 245 to cause the OLED panels 223 and 243 to display an image. The display control unit 151 controls the timing at which the OLED drive circuits 225 and 245 draw on the OLED panels 223 and 243 based on the signal output from the image processing unit 145, controls the luminance of the OLED panels 223 and 243, and the like.

  In addition, the display control unit 153 displays input content candidates corresponding to at least one of the movement of the fingers and the shape and position detected by the detection control unit 151 in the display area 310 described later. The display control unit 151 compares the finger movement, shape, and position detected by the detection control unit 151 with information for specifying a gesture or a hand sign registered in the pattern registration dictionary 125, and at least a part of them matches. Select the gesture or hand sign you want to use. The display control unit 153 displays a character image indicated by the character information associated with the selected gesture or hand sign as a candidate for input content.

  The input content candidates displayed by the display control unit 153 are not limited to characters and symbols. For example, it is assumed that the display control unit 153 displays the image of the software keyboard in the display area 310, and the detection control unit 151 detects the position of the finger from the captured image data and detects the selected key. In this case, the display control unit 153 may display the key displayed at the position of the finger detected by the detection control unit 151 as a candidate for input content.

  The HMD 100 may include an interface (not shown) for connecting various external devices that are content supply sources. For example, an interface corresponding to a wired connection such as a USB interface, a micro USB interface, or a memory card interface may be used, or a wireless communication interface may be used. The external device in this case is an image supply device that supplies an image to the HMD 100, and a personal computer (PC), a mobile phone terminal, a portable game machine, or the like is used. In this case, the HMD 100 can output images and sounds based on content data input from these external devices.

FIG. 6 is a flowchart showing the operation of the control unit 150. In particular, FIG. 6 is a flowchart showing an operation of detecting a user's gesture and hand sign and inputting characters and symbols corresponding to the detected gesture and hand sign.
First, the control unit 150 determines whether or not a preset operation has been accepted (step S1). The preset operation includes, for example, the operation of the button 11, the up / down key 15, and the track pad 14.
In addition, the control unit 150 detects the movement of the user's head with the 6-axis sensor 235 mounted on the image display unit 20, and is set in advance when a preset movement of the head is detected. It may be determined that the operation has been accepted. For example, the control unit 150 may determine that a preset operation has been received when a head movement such as nodding or swinging is detected based on the detection value of the 6-axis sensor 235.
In addition, the control unit 150 detects the user's finger area from the captured image data captured by the camera 61, and determines that a preset operation has been accepted when a preset gesture or hand sign is detected. May be.
Further, the control unit 150 may determine whether or not a preset operation has been received based on an audio signal input from the microphone 63. The control unit 150 compares the audio signal input from the microphone 63 with the audio pattern acquired from the audio pattern dictionary 126. The controller 150 may determine that a preset operation has been accepted when an audio signal that matches a finger snap or a knock sound registered in the audio pattern dictionary 126 is input.

  When it is determined that the preset operation has not been accepted (step S1 / NO), the control unit 150 waits until the preset operation is accepted. Execute the process. When it is determined that a preset operation has been received (step S1 / YES), the control unit 150 displays the detection range image 320 on the display area 310 (step S2), and the user's finger as an indicator is displayed. Start detection. The detection range image 320 and the display area 310 will be described with reference to FIG. The detection range image 320 corresponds to the “image indicating the detection state” of the present invention, and also corresponds to the “image indicating the detection area”.

FIG. 7 is a diagram illustrating an example of the field of view 300 of the user wearing the HMD 100.
The HMD 100 causes the image light L of the internally processed image and the external light OL to overlap and enter the user's eyes. For this reason, the user can see the outside scene through the right light guide plate 26 and the left light guide plate 28, and the image by the image light L is visually recognized over the outside scene. The outside scene is a scene that can be seen through the image display unit 20 by the user's eyes. FIG. 7 shows a lesson scene in which a teacher conducts a lesson using a blackboard as an outside scene. That is, FIG. 7 shows a field of view 300 of a student wearing the HMD 100.
The HMD 100 displays an image based on the image light L (hereinafter referred to as a display image) in the display area 310. A display area 310 indicates a range in which the HMD 100 can display a display image. In the display area 310, the user visually recognizes a display image displayed on the HMD 100 and an outside scene. In addition, the user visually recognizes only the outside scene outside the display area 310. In FIG. 7, a rectangular image showing the display area 310 is shown for explanation, but the user does not visually recognize the image showing the display area 310.

  FIG. 7 shows a state where the detection range image 320 is displayed as a display image. The detection range image 320 is an image indicating a range in which the control unit 150 detects the user's finger. The control unit 150 extracts data in a range corresponding to the detection range image 320 from the captured image data of the camera 61, and detects a finger region from the extracted range data. By displaying the detection range image 320 in the display area 310, it is not necessary to detect a finger from the entire display area 310, and the detection time of the finger area can be shortened. The detection range image 320 includes a left detection range image 321 indicating the detection range of the user's left hand finger and a right detection range image 322 indicating the detection range of the user's right hand finger.

  When the control unit 150 detects a finger region from the range corresponding to the left detection range image 321 of the captured image data, the control unit 150 determines the detected finger region as the finger of the left hand of the user. In addition, when the control unit 150 detects a finger region from the range corresponding to the right detection range image 322 of the captured image data, the control unit 150 determines the detected finger region as the finger of the right hand of the user. The control unit 150 does not need to determine whether the detected finger area is the right hand finger area or the left hand finger area, and thus the time required for the finger recognition process can be reduced. In addition, it is possible to reduce the occurrence probability of erroneous recognition that erroneously recognizes the left hand and the right hand.

When the detection range image 320 is displayed in the display area 310, the user performs a hand sign or gesture corresponding to the character or symbol desired to be input with his / her fingers. At this time, the user performs a hand sign or gesture so that the left and right fingers are displayed in the left detection range image 321 and the right detection range image 322, respectively.
The user's hand sign or gesture may be one in which the left and right fingers each represent one letter or symbol, the left hand indicates a vowel, the right hand indicates a consonant, etc. It may represent one character.
FIG. 8 is a diagram illustrating a state in which the left hand is within the frame of the left detection range image 321 and the right hand is within the frame of the right detection range image 322.

  Next, the control unit 150 causes the camera 61 to perform imaging and acquires captured image data generated by the camera 61 (step S3). The control unit 150 extracts a region corresponding to the detection range image 320 from the acquired captured image data. The control unit 150 uses a parameter generated by calibration to specify a range corresponding to the detection range image 320 of the captured image data, and extracts data in the specified range. The control device 10 performs, in advance, calibration for associating the position on the captured image data of the camera 61 with the position on the display area 310 where the image display unit 20 displays an image before starting this processing flow. Run. In the present embodiment, calibration is performed by the following method.

  The control unit 150 recognizes and tracks the reference real object corresponding to the calibration image from the captured image data of the camera 61 while displaying the calibration image on the image display unit 20. In this state, the user moves the position of the calibration image on the image display unit 20 via the user interface such as the track pad 14. Then, when the user perceives that the calibration image and the reference real object overlap (at least one of position, size, and orientation substantially coincides), the user is notified to the control unit 150 via the user interface. To do. In accordance with the acquired notification, the control unit 150 associates the detection result of the reference real object at the timing (position on the captured image data) with the display position of the calibration image on the image display unit 20. Is generated.

  Next, when the control unit 150 extracts data in a range corresponding to the detection range image 320 from the captured image data, the control unit 150 executes any known technique on the extracted data and executes detection processing for detecting a finger region. (Step S4). For example, the control unit 150 extracts the skin color area from the extracted captured image data area by the skin color extraction process, compares the pixel value of the extracted skin color area with the threshold value by the threshold process, and performs a finger operation. Find the area. Further, the control unit 150 extracts the contour of the obtained finger region by the edge detection process, and detects the finger by using feature point matching or curvature determination based on the contour unevenness.

  In addition, the control unit 150 may obtain a difference between a plurality of consecutive captured image data, identify a region (moving region) with movement based on the obtained difference, and extract the identified moving region as a finger region. . In addition, the control unit 150 may detect a finger region using a technique similar to that generally used for face detection, such as Boosting or SVM (Support Vector Machine).

  Next, the control unit 150 determines whether or not a finger region is detected by the detection process (step S5). When the finger region is not detected (step S5 / NO), the control unit 150 proceeds to the process of step S7. Further, when detecting the finger region (step S5 / YES), the control unit 150 causes the storage unit 122 to store captured image data (hereinafter referred to as finger image data) of the detected finger region (step S6).

  The control unit 150 determines whether or not the detection process has been executed for a preset number of captured image data (step S7). The preset number of captured image data may be captured image data captured continuously, or captured image data extracted every predetermined number of captured image data captured continuously. May be. Note that captured image data for one frame image obtained by imaging by the camera 61 is represented as one sheet. When the detection process is not executed for the preset number of captured image data (step S7 / NO), the control unit 150 returns to step S3 and newly acquires captured image data from the storage unit 122. And the control part 150 performs a detection process with respect to the acquired captured image data (step S4).

  In addition, when the detection process is performed on a predetermined number of pieces of captured image data (step S7 / YES), the control unit 150 acquires the finger image data stored in the storage unit 122 in step S6. Then, when there are a plurality of pieces of acquired finger image data, the control unit 150 compares these finger image data and determines whether or not there is a change in the position and shape of the fingers (step S8). For example, the control unit 150 determines whether or not there is a movement of the finger when the finger is detected in the detection range image 320 as the start of the gesture and when the finger comes out from the detection range image 320 as the end of the gesture. Determine whether or not. In addition, the control unit 150 may determine whether or not there is a movement of the finger with the time when the movement of the finger is detected as the start time of the gesture and the time when the movement of the finger stops as the time when the gesture ends.

  When it is determined that there is a change in the shape or position of the finger, that is, there is a movement of the finger (step S8 / YES), the control unit 150 determines that the operation performed by the user is a gesture. In this case, the control unit 150 determines a gesture based on the state of the finger detected from the captured image data and the gesture registration dictionary 1251, and input candidate characters and symbols corresponding to the determined gesture (hereinafter collectively referred to as “general”). The candidate character 335 is selected.

  The control unit 150 matches the information specifying the gesture registered in the gesture registration dictionary 1251 with the detected finger state, and selects a gesture that satisfies the matching result (step S9). For example, the control unit 150 selects a gesture in which at least a part of each state included in the detected finger movement matches the information specifying the gesture registered in the gesture registration dictionary 1251 (step S9). One or more gestures may be selected here. The control unit 150 displays an image indicating the selected gesture in the candidate display area 330, and displays the character associated with the gesture as the candidate character 335 in the candidate display area 330 (step S10). The candidate display area 330 is an area for displaying the candidate character 335 determined by the control unit 150 based on the gesture registration dictionary 1251 and the hand sign registration dictionary 1252. The candidate display area 330 is displayed in the display area 310 in correspondence with the position where the finger of the user who is the indicator is visually recognized. The candidate display area 330 corresponds to the “image showing the detection state” of the present invention, and also corresponds to the “candidate display area” of the present invention.

  In addition, when it is determined that there is no change in the position or shape of the finger, that is, there is no movement of the finger (step S8 / NO), the control unit 150 refers to the hand sign registration dictionary 1252 and selects the candidate character 335. To do. The control unit 150 compares the detected finger shape and position with information for specifying the hand sign, and selects a hand sign that at least partially matches the detected finger shape and position (step S11). . The hand sign selected here may be one or plural. When selecting the hand sign, the control unit 150 displays an image indicating the selected hand sign and the candidate character 335 associated with the hand sign in the candidate display area 330 (step S12).

Further, the control unit 150 determines the hand sign based on the detected finger shape and position when it is determined that the finger shape is the same shape continuously for a preset time or longer. Candidates may be displayed. Increasing the probability that the candidate character 335 intended by the user is displayed in the candidate display area 330 by selecting the hand sign when the shape of the same finger is continuously detected for a preset time or longer. Can do.
In addition, when it is determined that the finger is in a stationary state for a predetermined time or longer, the control unit 150 detects the shape and position of the finger, and the hand sign registration dictionary 1252 based on the detected shape and position of the finger The hand sign may be determined with reference to FIG. The state where the finger is kept stationary for a predetermined time or more corresponds to “when the indicator meets a preset condition” of the present invention.
In addition, the control unit 150 displays a predetermined symbol or the like in the display area 310, and detects a hand sign when a finger passes the symbol display position or overlaps the symbol display position. Also good. The timing at which the control unit 150 detects the hand sign may be, for example, when the position of the user's finger overlaps the symbol display position, or the movement of the finger after the symbol display position passes. It may be a stationary timing. When the finger passes the symbol display position, this corresponds to “a case where the position of the indicator overlaps a preset position” according to the present invention.

FIG. 9 is a diagram showing a candidate display area 330.
The candidate display area 330 is displayed at a position corresponding to the detection range image 320 or a position corresponding to a finger detected in the detection range image 320. The candidate display area 330 includes a left candidate display area 331 and a right candidate display area 332. The left candidate display area 331 and the right candidate display area 332 correspond to the “candidate display area” of the present invention. The left candidate display area 331 is displayed at a position corresponding to the left detection range image 321 or a position corresponding to a finger detected in the left detection range image 321. The right candidate display area 332 is displayed at a position corresponding to the right detection range image 322 or a position corresponding to the finger detected in the right detection range image 322. In the left candidate display area 331, a candidate character 335 (hereinafter referred to as the left candidate character 335A) determined based on the shape, position, and movement of the left hand detected by the control unit 150 from the captured image data is displayed. In the right candidate display area 332, a candidate character 335 (hereinafter referred to as a right candidate character 335B) determined based on the shape, position, and movement of the right hand detected by the control unit 150 from the captured image data is displayed. When the left candidate character 335A and the right candidate character 335B are collectively referred to as a candidate character 335.

FIG. 9 shows a display example in which the left candidate display area 331 is displayed around the left detection range image 321 and the right candidate display area 332 is displayed around the right detection range image 322.
The periphery of the left detection range image 321 is a range in the vicinity of the left detection range image 321 that can be recognized by the user when the left candidate display area 331 and the left detection range image 321 are associated with each other. In addition, the control unit 150 may display the left candidate display area 331 around the left detection range image 321 based on the distance set in advance by the user. The same applies to the distance between the right candidate display area 332 and the right detection range image 322.

The display positions of the left candidate display area 331 and the right candidate display area 332 are not limited to the example shown in FIG. For example, the left candidate display area 331 may be displayed in contact with the left detection range image 321, and the right candidate display area 332 may be displayed in contact with the right detection range image 322.
The left candidate display area 331 is displayed so that at least a part of the left candidate display area 331 overlaps the left detection range image 321, and the right candidate display area 332 is displayed so that at least a part of the left candidate display area 332 overlaps the right detection range image 322. The candidate display area 332 may be displayed. Further, as the position corresponding to the detected finger, for example, the left candidate display area 331 and the right candidate display area 332 may be displayed at the position of a predetermined fingertip of the finger of the user's finger, for example. Good. By displaying the candidate display area 330 in the vicinity of the detection range image 320 in this way, the user can select the candidate character 335 displayed in the vicinity of the finger without moving the line of sight.

With reference to the flowchart shown in FIG. 10, the character input operation based on gesture and hand sign detection will be described.
Next, the control unit 150 detects an operation of selecting the left candidate character 335A displayed in the left candidate display area 331 and an operation of selecting the right candidate character 335B displayed in the right detection range image 322. The control unit 150 causes the camera 61 to perform imaging and acquires captured image data generated by the camera 61 (step S13). The control unit 150 uses the parameters generated by the calibration to specify the range of the captured image data in which the candidate display area 330 is captured, and executes any known technique on the captured image data in the specified range. Then, detection processing for detecting a finger region is executed. When executing the detection process, the control unit 150 determines whether or not a finger region has been detected from the captured image data (step S14). When the finger region cannot be detected (step S14 / NO), the control unit 150 returns to step S13, acquires captured image data (step S13), and executes the detection process again.

  In addition, when a finger region is detected (step S14 / YES), the control unit 150 specifies the selected candidate character 335 based on the region of the captured image data in which the finger region is detected. The control unit 150 detects a finger area for each of the left candidate display area 331 and the right candidate display area 332. When a finger region is detected in the left candidate display region 331, the control unit 150 determines that the candidate character 335 displayed in the captured image data range in which the finger region is detected is the left candidate character 335A (step S15). Further, when a finger region is detected in the right candidate display region 332, the control unit 150 determines that the candidate character 335 displayed in the captured image data range in which the finger region is detected is the right candidate character 335B (step S15). ). The control unit 150 stores the candidate character 335 determined to be selected in the storage unit 122 as the selected character (step S16).

  Next, the control unit 150 acquires the captured image data of the camera 61 (step S17), executes a detection process on the region of the captured image data corresponding to the detection range image 320 (step S18), and the finger region. Is detected. When the finger region is detected (step S19 / YES), the control unit 150 returns to step S6 shown in FIG. 6 and stores the finger image data in the storage unit 122. Thereafter, the control unit 150 repeats the processes after step S7. Further, when the finger region is not detected (step S19 / NO), the control unit 150 determines whether or not a predetermined time has elapsed since the previous finger region detection (step S20). When the predetermined time has not elapsed since the previous finger region detection (step S20 / NO), the control unit 150 returns to step S17, acquires captured image data, and executes a detection process (step S17).

  In addition, when a predetermined time has elapsed since the previous detection of the finger region (step S20 / YES), the control unit 150 reads the selected character from the storage unit 122. Here, the selected character that the control unit 150 reads from the storage unit 122 may be one character or a plurality of characters, that is, a character string. When the selected character is read, the control unit 150 executes a corresponding process based on the read selected character (step S21). For example, when a web browser is activated as an application program, the control unit 150 inputs a selected character in the search field of the web browser. In addition, when the mail application is activated, the control unit 150 inputs a selected character in, for example, a mail destination or text.

In the flowchart described above, in step S15, the candidate character 335 displayed in the candidate display area 330 in which the finger area is detected is determined as the selected character.
As another operation, for example, the control unit 150 may assign a number to each of the plurality of candidate characters 335 and display the numbers together with the candidate characters 335 in the candidate display area 330. The user creates a finger character having a number corresponding to the candidate character 335 to be selected in the detection range image 320. The control unit 150 identifies the shape of the finger based on the captured image data, and determines the selected candidate character 335. Further, the user may select a candidate character 335 by performing a gesture corresponding to the number assigned to the candidate character 335 to be selected.

In FIG. 7, the left detection range image 321 and the right detection range image 322 are shown as the detection range image 320. However, one detection range image 320 may be displayed on the display area 310. In this case, the control unit 150 detects the finger area of both hands of the user from the range corresponding to the detection range image 320 of the captured image data.
In FIG. 7, circular images are displayed as the left detection range image 321 and the right detection range image 322, but the shape of the detection range image 320 is not limited to a circle. For example, the detection range image 320 may be a rectangular image or a polygonal image.
The size of the detection range image 320 is set in advance so as to be larger than the size of the user's finger captured in the captured image data captured by the camera 61.

  The timing at which the left detection range image 321 and the right detection range image 322 are displayed in the display area 310 is the timing at which the image of the user's hand (either left hand, right hand, or both hands) is detected from the captured image data. Also good. When the user's finger (either the left hand, the right hand, or both hands) is detected from the captured image data of the camera 61, the control unit 150 displays the left detection range image 321 and the right detection range image 322 corresponding to the detected finger. At least one is displayed in the display area 310. That is, when the left hand of the user is detected, the control unit 150 displays the left detection range image 321 such that the detected left hand finger is included in the range of the left detection range image 321. Similarly, when the right hand of the user is detected, the control unit 150 displays the right detection range image 322 such that the detected right hand finger is included in the range of the right detection range image 322.

Further, the control unit 150 may track the detected movement of the finger and change the display position and display range of the left detection range image 321 and the right detection range image 322 in accordance with the movement of the finger.
The control unit 150 detects a finger from captured image data sequentially captured by the camera 61, and when a change occurs in the position of the detected finger, the left detection range image 321 or the right detection range corresponding to the finger whose position has changed. The display display and display range of the image 322 are changed.
That is, when determining that the position of the left hand of the user has changed, the control unit 150 causes the left detection range image 321 to follow the change in the position of the left hand, and the left hand is included in the range of the left detection range image 321. In this way, the display position of the left detection range image 321 is changed. Further, the control unit 150 may change the display range of the left detection range image 321 according to the change in the position of the left hand.
If the control unit 150 determines that a change has occurred in the position of the right hand of the user, the control unit 150 causes the right detection range image 322 to follow the change in the position of the right hand, and the right hand is included in the range of the right detection range image 322. In this way, the display position of the right detection range image 322 is changed. Further, the control unit 150 may change the display range of the right detection range image 322 in accordance with the change in the position of the right hand.
By causing the left detection range image 321 and the right detection range image 322 to follow the movement of the user's finger, the user does not need to perform a gesture or hand sign at a fixed position, and can perform a gesture or hand sign more freely. It can be performed.

FIG. 11 is a diagram illustrating another display example of the candidate display area 330.
In FIG. 11, the left candidate display areas 331 are displayed in directions corresponding to the four directions of the upper, lower, left, and right directions of the left detection range image 321, and the right candidates are displayed in the directions corresponding to the upper, lower, left, and right directions of the right detection range image 322. The example which each displayed the area | region 332 is shown.
The user moves his / her finger in the direction of the candidate display area 330 where the candidate character 335 desired to be input is displayed. The control unit 150 acquires captured image data of the camera 61 and detects a finger region from each region of the captured image data corresponding to the upper, lower, left, and right candidate display regions 330 by the method described above. When the finger area is detected, the control unit 150 determines that the candidate character 335 displayed in the candidate display area 330 that has detected the finger area is selected. Since the user only has to move his / her finger in the direction in which the character to be input is displayed, the candidate character 335 can be easily selected.
FIG. 11 shows a case in which the candidate display area 330 is displayed in four directions, top, bottom, left, and right of the detection range image 320. The direction in which the candidate display area 330 is displayed according to the number of candidate characters 335 to be selected is shown. It may be changed. For example, when two characters are selected as the candidate characters 335, the candidate display area 330 can be displayed in the vertical direction or the horizontal direction of the detection range image 320.

  In addition, the control unit 150 may store the candidate character 335 and the selected character actually selected by the user in the storage unit 122 in association with each other. When the candidate character 335 determined based on the user's gesture or hand sign is the candidate character 335 stored in the storage unit 122, the control unit 150 displays the selected character associated with the candidate character 335 in the display area 310. To display. As described above, by recognizing the candidate character 335 and the selected character and storing them in the storage unit 122, it is possible to improve the recognition accuracy of gestures and hand signs. Further, even if the hand or finger shape created by the user has a wrinkle, the candidate of characters and symbols intended by the user can be displayed in the candidate display area 330.

FIG. 12 is a diagram illustrating another display example of the candidate character 335.
The display example shown in FIG. 12 displays a plurality of candidate characters 335 as candidates in the circumferential direction of an image showing a ring (hereinafter referred to as a ring image 325), and hand signs indicating the plurality of candidate characters 335. . In the display example shown in FIG. 12, candidate characters 335 are displayed at positions corresponding to the four directions of the ring image 325 in the vertical and horizontal directions.

When the user selects one candidate character 335 from among the plurality of candidate characters 335, for example, the user performs the following operation.
For example, when selecting the right candidate character 335B of the right hand, the user performs a gesture of moving the right hand from left to right or from right to left, and moves the right candidate character 335B to be selected to the selected position. The selection position is a preset position, and may be any position corresponding to the four directions of the top, bottom, left, and right of the ring image 325. In the present embodiment, a case where the position corresponding to the downward direction of the ring image 325 is the selection position will be described. In FIG. 12, the position where “B” is displayed as the right candidate character 335B corresponds to the selected position.

The control unit 150 detects the movement of the right hand from the imaged image data of the camera 61, rotates the ring image 325 in the direction in which the right hand moves, and also rotates a plurality of candidate characters 335 as the ring image 325 rotates.
When the candidate character 335 is displayed at the selected position, the user performs a predetermined confirming operation to confirm the candidate character 335. The confirmation operation may be, for example, a nodding operation, that is, an operation of tilting the user's head forward in the direction of the user's line of sight. The control unit 150 detects the movement of the user's head based on the detection value of the 6-axis sensor 235, and determines whether or not a confirmation operation has been input. Further, for example, the user may utter a sound “confirmed”. Based on the audio signal input from microphone 63, control unit 150 determines whether or not the user has uttered “confirmed”.

  In addition, when the user selects one candidate character 335 from among the plurality of candidate characters 335, the user may select the candidate character 335 by placing a finger on the position where the candidate character 335 to be selected is displayed. . That is, when selecting the right candidate character 335B, the user overlaps the position of the right candidate character 335B for selecting the right hand, and when selecting the left candidate character 335A, the user overlaps the position of the left candidate character 335A for selecting the left hand.

  In the description of the flowchart shown in FIG. 6, the control unit 150 displays the left detection range image 321 and the right detection range image 322 in the display area 310 when a preset operation is received. As another operation, the control unit 150 may display the left detection range image 321 and the right detection range image 322 in the display area 310 when executing a preset program. The preset program is, for example, an application program (hereinafter referred to as an application) that requires character input, such as browser software or word processor software.

FIG. 13 is a diagram showing the software keyboard 351 and the text input field 352 displayed in the display area 310.
When executing an application that requires character input, the control unit 150 displays the software keyboard or the left detection range image 321 and the right detection range image 322 in the display area 310 according to the setting data 123. In addition, when a user's finger is detected at a position corresponding to the text input field 352, the control unit 150 displays the software keyboard or the left detection range image 321 and the right detection range image 322 according to the setting data 123 as a display area. 310 may be displayed. Further, when the left detection range image 321 and the right detection range image 322 are displayed in the display area 310, the control unit 150 shifts to a state of detecting an indicator and starts detecting the user's finger. A state where an application that requires character input is selected by the user and execution of the selected application is started by the control unit 150 corresponds to “when a preset operation is detected” of the present invention. A case where the user's finger is detected at a position corresponding to the text input field 352 is also equivalent to “a case where a preset operation is detected” in the present invention.

The setting data 123 stored in the storage unit 122 includes information that sets whether to perform character input by the software keyboard 351 or character input by gesture or hand sign when executing an application that requires character input. It is.
The setting data 123 can be rewritten by operating the operation unit 110, for example. The control unit 150 changes the setting data 123 according to the operation received by the operation unit 110. When the setting data 123 is changed, the character input is changed from the software keyboard to the gesture or the hand sign, or the gesture or hand sign is changed to the software keyboard.

  In addition, for each application that requires character input, it may be set whether character input is performed using a software keyboard or character input is performed using a gesture or a hand sign. That is, the setting data 123 is generated for each application that requires character input. When an application execution instruction is input, the control unit 150 displays the software keyboard 351 or the left detection range image 321 and the right detection range image 322 in the display area 310 according to the setting data 123 corresponding to the application.

  Moreover, although embodiment mentioned above demonstrated the input of the character by a hand sign or a gesture, the input by a hand sign or a gesture is not limited only to a character. For example, the function of the HMD 100 can be operated by a hand sign or a gesture, or an external device connected to the HMD 100 can be operated.

A case where the audio function of the HMD 100 is operated by a hand sign or a gesture will be described.
For example, the hand sign and gesture of the user's left hand are associated with operations for selecting setting items such as volume control and music selection. Further, the hand sign or gesture of the right hand is associated with the operation for setting the selected setting item.
In this case, in the gesture registration dictionary 1251, as left hand information, information specifying the left hand gesture and information indicating setting items (for example, volume control and music selection) associated with each gesture are registered in association with each other. Is done.
In the gesture registration dictionary 1251, information specifying the right hand gesture and information indicating an operation item associated with the gesture are registered for each setting item as right hand information. The hand sign registration dictionary 1252 has the same configuration. For example, when the setting item is volume control, the information on the right hand includes information for specifying a gesture and operation items (for example, volume up, volume down, mute) associated with the gesture. When the setting item is song selection, information specifying the gesture as the right hand information and operation items (for example, playback, pause, stop) associated with the gesture are volume control information. It is stored in the gesture registration dictionary 1251.

FIG. 14 is a diagram illustrating a display image displayed on the display image 310 when the function of the HMD 100 is operated by a hand sign or a gesture. In particular, FIG. 14 is a diagram illustrating a display image displayed on the display image 310 when the audio function of the HMD 100 is operated.
First, the control unit 150 detects a hand sign or gesture of the left hand based on the captured image data of the camera 61. When the left hand hand sign or gesture is detected, the control unit 150 displays setting items corresponding to the detected left hand hand sign or gesture in the left display area 361. FIG. 14 shows a state in which volume control is displayed as a setting item in the left display area 361. The setting items displayed in the left display area 361 correspond to the “first operation image” of the present invention.

Next, the control unit 150 detects a confirming operation or a canceling operation. The confirmation operation is an operation for confirming the setting item displayed in the left display area 361. The cancel operation is an operation for canceling the setting item displayed in the display area 361. When the confirmation operation is detected, the control unit 150 detects a right hand hand sign or gesture based on the captured image data.
In addition, when a cancel operation is detected, the control unit 150 deletes the display of the setting item displayed in the left display area 361 and detects the left hand hand sign or gesture again.
Further, the confirmation operation and the cancellation operation may be, for example, an operation by a hand sign or a gesture of the left hand, or a head movement detected by the 6-axis sensor 235.

Further, when a confirmation operation is detected, the control unit 150 detects a right hand hand sign or gesture based on the captured image data. The control unit 150 displays an operation item corresponding to the detected hand sign or gesture in the right display area 362. FIG. 14 shows a state in which the volume is displayed as an operation item in the right display area 362. The operation item displayed in the display area 362 corresponds to the “second operation image” of the present invention.
Thereafter, the control unit 150 detects a confirming operation or a canceling operation as in the case of the hand sign or gesture of the left hand. When the confirmation operation is detected, the control unit 150 executes processing corresponding to the operation item displayed in the right display area 362. In addition, when a cancel operation is detected, the control unit 150 erases the display of the operation item displayed in the right display area 362 and detects the right hand hand sign or gesture again.

Further, when determining the setting item corresponding to the hand sign or gesture of the left hand, the control unit 150 may display all the operation items of the right hand associated with the determined setting item in the right display area 362.
For example, when it is determined that the volume control is selected as the setting item, the control unit 150 displays volume up, volume down, and mute in the right display area 362. The user moves the right hand so that the operation item to be operated overlaps with the displayed position. The control unit 150 detects the position of the right hand based on the captured image data, and determines the operation item displayed at the detected position. The control unit 150 executes processing corresponding to the determined operation item.

FIG. 15 is a diagram illustrating a display image displayed in the display area 310 when an external device connected to the HMD 100 is operated by a hand sign or a gesture. In particular, FIG. 15 is a diagram illustrating a display image when an electric knife as an external device is operated by a hand sign or a gesture.
The HMD 100 is connected to a control device (not shown) that controls the current flowing through the electric knife through the communication unit 117. The connection between the control device and the HMD 100 may be wireless or may be connected by a wired cable.

  First, the control unit 150 detects a predetermined mark from the captured image data of the camera 61. For example, a predetermined mark such as a barcode or a QR code (registered trademark) that identifies the external device is attached to the external device such as an electric knife. The control unit 150 detects the predetermined mark from the captured image data, and specifies the external device held in the user's hand (here, it is assumed that it is the right hand).

When the control unit 150 detects the predetermined mark of the captured image data and specifies the external device, the control unit 150 displays the setting item of the specified external device in the right display area 362.
Here, the hand sign or gesture of the user's right hand is associated with an operation for selecting a setting item such as a switch or current adjustment.
In this case, in the gesture registration dictionary 1251, information specifying the right hand gesture and information indicating setting items (for example, a power switch and current adjustment) associated with each gesture are associated with each other as the right hand information. be registered.
In the gesture registration dictionary 1251, information specifying the left hand gesture and information indicating an operation item associated with the gesture are registered for each setting item as left hand information. The hand sign registration dictionary 1252 has the same configuration. For example, if the setting item is current adjustment, information specifying the gesture as left hand information and an operation item (for example, current up or current down) associated with the gesture are registered as current adjustment information. It is stored in the dictionary 1251.

When the control unit 150 detects a predetermined mark and specifies the external device held by the right hand of the user, the control unit 150 displays the setting item of the specified external device in the right display area 362.
When the setting item is displayed in the right display area 362, the user selects one of the setting items displayed in the right display area 362 by operating the right hand or the head. The right-hand operation is an operation of placing the right hand on the position where the setting item to be selected is displayed. The head operation is, for example, an operation of turning the head in the direction in which the setting item is displayed.

  When a setting item is selected, the control unit 150 displays an operation item associated with the selected setting item in the left display area 361. For example, when current adjustment is selected as the setting item, the control unit 150 displays current up or current down as operation items in the left display area 361.

  When the operation item is displayed in the left display area 361, the user operates the operation item by operating the left hand or the head. For example, an operation of placing the left hand on the position where the operation item to be selected is displayed may be a gesture or a hand sign. The head operation is, for example, an operation of turning the head in the direction in which the operation item is displayed.

When an operation item is selected, the control unit 150 transmits control information including at least one of first operation information corresponding to the selected operation item and second operation information corresponding to the right-hand operation to the communication unit 117. To the control device. The first operation information includes, for example, information for instructing current up and current down, and information for instructing to turn on and off the electric knife.
The control information transmitted to the control device may be information including the first operation information and the second operation information. The second operation information includes, for example, identification information for identifying the electric knife held by the user.
When the user is left-handed and has an electric knife in his left hand, control information including second operation information is transmitted from the HMD 100 to the control device. The second operation information in this case is information for instructing flow-up and current-down, and information for instructing power-on and power-off of the electric knife.
The first operation information corresponds to “first information” of the present invention, and the second operation information corresponds to “second information” of the present invention.

As described above, the HMD 100 according to the first embodiment includes the image display unit 20, the camera 61 and the imaging control unit 147 that function as a detection unit, and the display control unit 153.
The image display unit 20 displays an image in the display area 310 so that the outside scene is transmitted and can be visually recognized together with the outside scene. The camera 61 and the imaging control unit 147 detect the user's finger as an indicator associated with the user's action in an area including at least a range that passes through the image display unit 20.
The display control unit 153 displays an image indicating a detection state in which the detection unit detects the user's finger on the display area 310, and the input content corresponding to at least one of the movement, shape, and position of the finger detected by the detection unit Display candidates. The image indicating the detection state includes a detection range image 320, a candidate display area 330, candidate characters 335 displayed in the candidate display area 330, a left display area 361, a right display area 362, and the like.
In the HMD 100 according to the first embodiment, since the image indicating the detection state of the indicator by the detection unit is displayed in the display area, the detection state of the finger by the detection unit can be easily confirmed. In addition, since the input content candidates corresponding to any of the movement, shape, and position of the finger detected by the detection unit are displayed at the position corresponding to the image indicating the detection area, the input accuracy with the finger is improved, and the user Usability can be improved.

In addition, the display control unit 153 displays an image indicating the detection state of the detection unit corresponding to the position where the finger is visually recognized in the outside scene.
Therefore, an image indicating the detection state of the detection unit is displayed in correspondence with the position where the finger is visually recognized, so that the user can confirm the detection state of the detection unit while inputting the operation by moving the finger.

In addition, the display control unit 153 displays the input content candidates corresponding to at least one of the movement, shape, and position of the finger detected by the detection unit at a position corresponding to the image indicating the detection state of the detection unit.
Therefore, the user can confirm the detection state by the detection unit while operating the indicator.

In addition, the display control unit 153 displays a detection range image 320 indicating the detection range on the display area 310 as an image indicating the detection state of the detection unit. In addition, the display control unit 153 displays input content candidates corresponding to at least one of the movement, shape, and position of the finger detected by the detection unit at a position corresponding to the detection range image 320 in the display area 310.
Accordingly, since the detection range image 320 indicating the detection area where the detection unit detects the finger is displayed as an image indicating the detection state of the detection unit, it is easy to confirm the detection area where the finger is detected and the operability is improved. Can be increased. In addition, since the input content candidates corresponding to at least one of the movement, shape, and position of the indicator detected by the detection unit are displayed at the position corresponding to the detection range image 320, the user operates the indicator while operating the indicator. The detection state of the detection unit can be confirmed.

In addition, the display control unit 153 displays the candidate character 335 in the candidate display area 330, and sets the candidate display area 330 to a position corresponding to the detection range image 320 or a position corresponding to the finger detected in the detection range image 320. Display.
Therefore, an operation of selecting a candidate from the candidate characters 335 displayed at the position corresponding to the detection range image 320 or the position corresponding to the finger detected in the detection range image 320 can be easily performed.

In addition, when the detection unit detects a finger at a position corresponding to the candidate display area 330, the display control unit 153 displays the candidate display area 330 based on at least one of the detected movement, shape, and position of the finger. Select one of a plurality of candidates.
Therefore, since any one of the plurality of candidate characters 335 can be selected by at least one of the movement, shape, and position of the fingers, the operation can be simplified.

In addition, when a finger is detected at a position corresponding to the candidate character 335 displayed in the display area 310, the display control unit 153 displays a candidate based on at least one of the detected movement, shape, and position of the finger. One of a plurality of candidate characters displayed in the area 330 is selected.
Therefore, any of a plurality of candidate characters can be selected based on the movement, shape, and position of the finger moved to the position corresponding to the candidate character 335.

The display control unit 153 displays a plurality of candidates, and selects a corresponding candidate when the moving direction of the finger detected by the detection unit corresponds to any direction in which the candidate is displayed.
Therefore, the candidate to be selected can be selected by moving the finger in the direction in which the candidate to be selected is displayed.

In addition, the display control unit 153 displays a candidate when the shape of the finger detected by the detection unit is the same shape for a preset time or more.
Therefore, candidates can be displayed by making the shape of the fingers the same shape for a preset time or longer, so that the candidates can be displayed by a simple operation.

Further, the display control unit 153 displays candidates when the position of the finger detected by the detection unit overlaps with a preset position.
Therefore, candidates can be displayed by a simple operation.

In addition, the display control unit 153 displays candidates when the shape of the indicator detected by the detection unit is a shape corresponding to a preset condition.
Therefore, candidates can be displayed by a simple operation.

In addition, the display control unit 153 displays a left detection range image 321 and a right detection range image 322, which are images indicating a detection area including the left hand of the user, as images indicating the detection area.
Accordingly, it is not necessary to determine whether the finger detected by the detection unit is the left hand or the right hand, and the probability that the candidate intended by the user is displayed can be increased.

The detection unit detects the left hand and the right hand of the user as indicators. The display control unit 153 includes a left display area 361 in which operation items to be operated by the left hand detected by the detection unit are displayed, and a right display area 362 in which operation items to be operated by the right hand are displayed. Is displayed in the display area.
Therefore, an operation on the operation item displayed in the left display area 361 can be performed with the left hand, and an operation on the operation item displayed in the right display area 362 can be performed with the right hand.

In addition, the HMD 100 transmits control information to the external device corresponding to at least one of the movement, shape, and position of the finger detected by the camera 61 as the detection unit.
Accordingly, the external device can be operated by at least one of the movement, shape, and position of the fingers.

The HMD 100 also includes a communication unit 117 that outputs control information to an external device in response to at least one of finger movement, shape, and position detected by the detection unit. The display control unit 153 includes control information including at least one of first operation information corresponding to the left hand operation detected by the detection unit and second operation information corresponding to the right hand operation detected by the detection unit. To the external device.
Therefore, the external device can be operated by the left hand operation and the right hand operation.

The HMD 100 also includes a microphone 63 as a voice input unit that inputs voice. The detection control unit 151 detects a finger when the microphone 63 inputs a preset voice.
Therefore, it is possible to cause the HMD 100 to start detecting fingers by voice input.

The HMD 100 includes a detection control unit 151 that detects an operation.
The detection control unit 151 shifts to a state of detecting a finger when a preset operation is detected.
Therefore, it is possible to shift to a state of detecting a finger by performing a preset operation.

In addition, the storage unit 122 stores a gesture registration dictionary 1251 in which a gesture indicating a preset movement trajectory of a user's finger is associated with a candidate. The storage unit 122 also stores a hand sign registration dictionary 1252 in which hand signs indicating preset shapes of fingers are associated with candidates.
The display control unit 153 selects a gesture or a hand sign by matching the finger movement or shape detected by the detection unit with the gesture registration dictionary 1251 or the hand sign registration dictionary 1252. The display control unit 153 displays candidates corresponding to the selected gesture or hand sign in the display area 310.
Therefore, it is possible to specify a gesture or hand sign based on the movement or shape of the user's finger and display candidates corresponding to the specified gesture or hand sign.

[Second Embodiment]
FIG. 16 is a diagram illustrating a usage state of the HMD 100 according to the second embodiment.
FIG. 16 shows a scene in which a conversation using text data is performed between a plurality of users wearing the HMD 100. In particular, FIG. 16 shows a scene where a user A and a user B have a conversation. Hereinafter, the HMD 100 worn by the user A is denoted as “HMD100A”, and the HMD100 worn by the user B is denoted as “HMD100B”. In addition, a function part (see FIG. 4) constituting “HMD100A” is denoted by “A”, and a function part (see FIG. 4) constituting “HMD100B” is denoted by “B”. explain.

User A and user B have a conversation while facing each other.
In a state where the user A and the user B face each other, the camera 61A of the HMD 100A captures not only gestures and hand signs performed by the user A but also gestures and hand signs performed by the user B. Similarly, in addition to the gestures and hand signs performed by the user B, the gestures and hand signs performed by the user A are also captured by the camera 61B of the HMD 100B.
It is assumed that the direction of the hand sign and gesture of the user A imaged by the camera 61A is the front. Since the user A and the user B face each other, the captured image data includes an image obtained by capturing the hand sign and gesture of the user B from the back side, which is a direction different from the front (opposite direction). .

  The control unit 150A of the HMD 100A detects the hand sign and gesture of the user A and the hand sign and gesture of the user B from the captured image data of the camera 61A. Further, the control unit 150B of the HMD 100B detects the hand sign and gesture of the user A and the hand sign and gesture of the user B from the captured image data of the camera 61B.

  The user A and the user B perform hand signs and gestures so that the fingers of the user A and the fingers of the user B do not overlap the cameras 61A and 61B. For example, the user A and the user B may not perform hand signs and gestures at the same time. Alternatively, the user A may make a hand sign or gesture on the left side when viewed from the user A, and the user B may make a hand sign or gesture on the left side when viewed from the user B.

FIG. 17 is a diagram showing the first detection range image 326 and the second detection range image 327 displayed in the display area 310.
Further, the user A adjusts the direction of the head so that the user B is within the angle of view of the camera 61A, and operates the operation unit 110 to cause the camera 61A to image the user B. At this time, the user A adjusts the orientation of the head so that the user B departs from the center of the captured image data and is captured on the right side or the left side of the captured image data.
The control unit 150A of the HMD 100A specifies a person area where a person is imaged from the imaged image data of the camera 61A, and a range including the specified person area is a detection range of hand signs and gestures performed by the user B who is a conversation partner. Set to. The controller 150A causes the display area 310 to display an image indicating the set detection range (hereinafter referred to as a first detection range image 326). In addition, the control unit 150A sets a detection range for detecting the hand sign or gesture of the user A in an area other than the first detection range image 326. The controller 150A causes the display area 310 to display an image (hereinafter, referred to as a twelfth outgoing range image) 327 indicating the set detection range. The first detection range image 326 corresponds to the first detection area of the present invention, and the second detection range image 327 corresponds to the second detection area of the present invention.

FIG. 18 is a diagram illustrating configurations of the control unit 150 and the storage unit 122 according to the second embodiment.
The storage unit 122A of the HMD 100A stores an external gesture registration dictionary 1253A and an external hand sign registration dictionary 1254A in addition to the gesture registration dictionary 1251A and the hand sign registration dictionary 1252A as the pattern registration dictionary 125A. The external gesture registration dictionary 1253A and the external hand sign registration dictionary 1254A correspond to “external dictionary information” of the present invention.
The external gesture registration dictionary 1253A is a dictionary in which information for specifying a gesture performed by a facing user (user B) and a candidate character 335 associated with the gesture are registered in association with each other. The external handsign registration dictionary 1254A is a dictionary in which information for specifying a handsign performed by the user (user B) and candidate characters 335 associated with the handsign are registered in association with each other. That is, the gesture registration dictionary 1251A and the hand sign registration dictionary 1252A are dictionaries for specifying hand signs and gestures captured from the front side. The external gesture registration dictionary 1253A and the external hand sign registration dictionary 1254A are dictionaries for specifying hand signs and gestures imaged from the back side that is the direction opposite to the front.

  150 A of control parts image the gesture and hand sign which the user B performed with the camera 61A. 150 A of control parts detect the finger area | region where the finger of the user B was imaged from the captured image data of the camera 61A, and specify the shape and position of the finger imaged by the detected finger area | region. The controller 150A identifies the hand sign or gesture performed by the user B with reference to the external gesture registration dictionary 1253A and the external hand sign registration dictionary 1254A based on the shape and position of the finger of the user B that has been identified.

The storage unit 122B of the HMD 100B also stores an external gesture registration dictionary 1253B and an external hand sign registration dictionary 1254B in addition to the gesture registration dictionary 1251B and the hand sign registration dictionary 1252B as the pattern registration dictionary 125B.
The external gesture registration dictionary 1253B is a dictionary in which information specifying a gesture performed by the user A facing the user B and a candidate character 335 associated with the gesture are registered in association with each other. The external hand sign registration dictionary 1254B is a dictionary in which information for specifying a hand sign performed by the user A facing the user B and a candidate character 335 associated with the hand sign are registered in association with each other. .

Also, before the users start a conversation, the HMDs 100 communicate with each other, and whether or not the external gesture registration dictionary 1253 and the external hand sign registration dictionary 1254 are stored in the storage unit 122 of the communication partner HMD 100 is determined. You may perform the process to confirm.
For example, it is assumed that the HMD 100A stores an external gesture registration dictionary 1253A and an external hand sign registration dictionary 1254A, and the HMD 100B does not store the external gesture registration dictionary 1253B and the external hand sign registration dictionary 1254B. In this case, before starting the conversation, the HMD 100A communicates with the HMD 100B and transmits the external gesture registration dictionary 1253A and the external hand sign registration dictionary 1254A to the HMD 100B. The HMD 100B stores the external gesture registration dictionary 1253A and the external hand sign registration dictionary 1254A received from the HMD 100A in the storage unit 122B. Further, the HMD 100B determines the gesture and hand sign of the user A using the dictionary received from the HMD 100A.

As described above, the HMD 100A of the second embodiment includes the storage unit 122A that stores the external gesture registration dictionary 1253A and the external hand sign registration dictionary 1254A.
The external gesture registration dictionary 1253A and the external hand sign registration dictionary 1254A are dictionaries for specifying gestures and hand signs that are at least one of the movement, shape, and position of the finger of the user B detected from different directions.
In the captured image data captured by the camera 61A of the HMD 100A, the finger of the user B captured from a direction different from the direction of the finger of the user A is captured. Based on the external gesture registration dictionary 1253A and the external hand sign registration dictionary 1254A, the control unit 150A specifies a hand sign or gesture by the finger of the user B captured from a direction different from the finger of the user A. In addition, the control unit 150A causes the display area 310 to display candidate characters 335 corresponding to the identified hand sign or gesture.
Therefore, it is not necessary for the user A and the user B to perform hand signs and gestures from the same direction, and it is possible to reduce time and labor when performing hand signs and gestures.

  Further, the HMD 100A includes a communication unit 117 that communicates with the HMD 100B that is an external head-mounted display device. The HMD 100A transmits the external gesture registration dictionary 1253A and the external hand sign registration dictionary 1254A to the HMD 100B through the communication unit 117. Therefore, the HMD 100B can detect at least one of the movement, shape, and position of a finger and display a candidate corresponding to the detection result.

  In addition, the user B imaged in the imaged image data of the camera 61 is detected, and the control unit 150A, based on the detection result, a first detection range image 326 that detects a gesture and a hand sign performed by the user A; A second detection range image 327 for detecting a gesture or a hand sign performed by the user B is displayed. Therefore, the control unit 150A does not need to determine whether the detected finger is the finger of the user A or the finger of the user B. For this reason, the time required for the finger recognition process can be shortened, and the probability of erroneously recognizing a hand sign or gesture by a finger can be reduced.

  In addition, in the first detection range image 326, candidates corresponding to hand signs and gestures captured in the captured image data are specified using the gesture registration dictionary 1251A and the hand sign registration dictionary 1252A. Further, in the second detection range image 327, candidates corresponding to hand signs and gestures captured in the captured image data are specified using the external gesture registration dictionary 1253A and the hand sign registration dictionary 1254A. Therefore, the hand sign and the gesture can be specified using the dictionary corresponding to the imaging direction in which the hand sign and the gesture are imaged, and the hand sign and the gesture can be specified with high accuracy.

[Third Embodiment]
This embodiment also assumes a scene in which the HMD 100 is worn by a plurality of users and text conversation is performed between the users as in the second embodiment described above. In the present embodiment, the hand sign and gesture performed by the user A are detected by the HMD 100A, and the hand sign and gesture performed by the user B are detected by the HMD 100B.
The control unit 150A of the HMD 100A captures the hand sign and gesture performed by the user A with the camera 61A, and specifies text corresponding to the hand sign and gesture input from the captured image data. Similarly, the control unit 150B of the HMD 100B captures a hand sign or gesture made by the user A with the camera 61B, and specifies text corresponding to the hand sign or gesture input from the captured image data. In the present embodiment, the user A and the user B do not need to face each other.

Further, the communication unit 117A of the HMD 100A worn by the user A and the communication unit 117B of the HMD 100B worn by the user B are connected by wireless communication such as Bluetooth or wireless LAN, for example. When the control unit 150A detects the hand sign or gesture made by the user A and specifies the text corresponding to the detected hand sign or gesture, the control unit 150A transmits the specified text data (text data) to the HMD 100B by the communication unit 117A. To do. When the HMD 100B receives the text data from the HMD 100A, the HMD 100B displays the received text data in the display area 310B.
Similarly, when the control unit 150B detects the hand sign or gesture made by the user B and specifies the text corresponding to the detected hand sign or gesture, the control unit 150B transmits the specified text data (text data) by the communication unit 117B. Send to HMD100A. When the HMD 100A receives the text data from the HMD 100B, the HMD 100A displays the received text data in the display area 310A.

FIG. 19 is a diagram showing a display screen 360B displayed in the display area 310B of the HMD 100B. In particular, FIG. 19 is a diagram showing a display screen 360B that displays the contents of conversations performed by the user A and the user B in time series. For example, on the left side of the display screen 360B, text indicating the utterance content of the user A is displayed in time series from top to bottom, and on the right side of the display screen 360B, text indicating the utterance content of the user B is displayed. Displayed in series from top to bottom.
A similar display screen 360A is also displayed in the display area 310A of the HMD 100A.

Information input by the user includes information that may be disclosed to other users and information that is not desired to be disclosed to other users such as a memo.
Based on the distance between the body (torso) of the user A and the hand, the control unit 150A determines whether the input text is information that is made public to other users or private information. .
For example, the control unit 150A compares the size of the hand or finger captured in the captured image data of the camera 61A with a preset threshold value, and determines whether the input character is disclosed or not disclosed Determine. When the size of the imaged hand or finger is equal to or less than a preset threshold value, control unit 150A determines that a hand sign or gesture has been performed at a position away from user A's body (torso). . In this case, the control unit 150A transmits the text data of the input text to the conversation partner HMD 100B. The control unit 150B of the HMD 100B displays the text data received from the HMD 100A on the display area 310B by the image display unit 20B.

  Further, the control unit 150A determines that a hand sign or gesture has been performed at a position close to the body (torso) of the user A when the size of the captured hand or finger is larger than a preset threshold value. To do. In this case, the control unit 150A does not transmit the text data of the input text to the conversation partner HMD 100B, but displays it in the display area 310A so that only the user A can visually recognize it.

  The threshold value used for comparison with the size of the hand or finger captured in the captured image data of the camera 61 may be set by calibration performed in advance. For example, the control unit 150A causes the image display unit 20A to display a message such as “Please do hand sign or gesture when inputting text to be disclosed” on the display area 310A. Thereafter, the controller 150A captures an image with the camera 61A, detects a finger region from the captured image data, and detects a preset size of a hand or a finger (for example, a thumb). In addition, the control unit 150A causes the image display unit 20A to display a message such as “please perform a hand sign or gesture when inputting non-public text” on the display area 310A. Thereafter, the controller 150A captures an image with the camera 61A, detects a finger region from the captured image data, and detects a preset size of a hand or a finger (for example, a thumb). Based on the size of the hand or finger captured by the camera 61 when the text is disclosed and the size of the hand or finger captured by the camera 61 when the text is not disclosed, the control unit 150A Is set to the optimum value.

The above-described embodiment is a preferred embodiment of the present invention. However, the present invention is not limited to this, and various modifications can be made without departing from the scope of the present invention.
For example, in the above-described embodiment, the user's finger is detected using the captured image data of the camera 61, but the method for detecting the user's finger is not limited to this method. For example, a camera that captures infrared rays is mounted on the HMD 100, and infrared rays output from a device attached to the user's hand are captured by this camera to detect the movement, shape, position, etc. of the user's fingers. Also good.
In addition, a sensor is attached to the user's finger and the movement, shape, position, etc. of the user's finger are detected by this sensor, and the movement, shape, position, etc. of the finger detected by the sensor are short-range wirelessly communicated. May be transmitted to the HMD 100.
In addition, an inertial sensor such as a gyro sensor or an acceleration sensor may be attached to the user's finger, and the motion, shape, position, etc. of the finger may be obtained by calculating backward from information measured as movement by the inertial sensor.

  In the above-described embodiment, the user's hand or finger shown in the captured image data of the camera 61 is detected. However, a finger including the user's arm may be detected. For example, the user expresses characters and symbols to be input to the HMD 100 by a hand flag signal. The control unit 150 of the HMD 100 detects a region where a finger including the arm is captured from the captured image data of the camera 61 and specifies the position of the arm captured in the detected region. Further, the control unit 150 detects the movement of the arm by detecting the region where the finger including the arm is captured from the captured image data continuously captured, and specifying the position of the arm captured in this region. Thus, the corresponding hand flag signal is specified, and the candidate character 335 is displayed in the candidate display area 330.

In the above description, the HMD 100 detects a gesture or a hand sign by using the operation of the button 11, the up / down key 15, the track pad 14, a preset head movement, or a preset voice input as a trigger. Explained the case of starting. As another trigger, when the HMD 100 is mounted on the user's head, the HMD 100 may start detecting a gesture or a hand sign.
For example, a mounting detection unit (not shown) is provided in the image display unit 20 of the HMD 100. For example, the mounting detection unit may detect that the HMD 100 is mounted on the user's head based on detection values of the 6-axis sensor 235 and the magnetic sensor 237 provided in the image display unit 20. In addition, a hard switch is provided in the image display unit 20 of the HMD 100, the mounting detection unit detects that the HMD 100 is mounted on the user's head and the state of the hard switch is switched, and the HMD 100 is mounted on the user's head. You may detect having mounted | worn.

  In the above-described embodiment, the finger display area is detected from the captured image data of the camera 61 and the candidate display area 330 is specified. As an operation other than this, the control unit 150 detects a finger region, specifies the corresponding candidate display region 330, and then blinks the candidate character 335 displayed in the specified region. And the control part 150 may determine with this candidate character 335 having been selected, when a motion of a user's head, for example, a nodding action, is detected. In addition, when a movement of the user swinging his / her head to the left or right is detected, it is preferable to acquire the captured image data of the camera 61 and detect the finger region again.

  In the above-described embodiment, the configuration in which the control device 10 is connected to the image display unit 20 by wire is illustrated. However, the present invention is not limited to this, and the image display unit 20 is wirelessly connected to the control device 10. It may be a configuration. In this case, the wireless communication method may be the method exemplified as the communication method supported by the communication unit 117, or may be another communication method.

  In addition, some functions of the control device 10 may be provided in the image display unit 20, and the control device 10 may be realized by a plurality of devices. That is, the control device 10 is not limited to the configuration including the box-shaped case 10A. For example, instead of the control device 10, a wearable device that can be attached to a user's body, clothes, or an accessory worn by the user may be used. The wearable device in this case may be, for example, a watch-type device, a ring-type device, a laser pointer, a mouse, an air mouse, a game controller, a pen-type device, or the like.

  Furthermore, in the above embodiment, the configuration in which the image display unit 20 and the control device 10 are separated and connected via the connection cable 40 has been described as an example. The present invention is not limited to this, and the control device 10 and the image display unit 20 may be configured integrally and mounted on the user's head.

  As the control device 10, a notebook computer, a tablet computer, or a desktop computer may be used. The control device 10 may be a game machine, a portable phone, a portable electronic device including a smartphone or a portable media player, other dedicated devices, or the like.

  Further, for example, instead of the image display unit 20, an image display unit of another method such as an image display unit worn like a hat may be adopted, and an image corresponding to the left eye LE of the user is displayed. What is necessary is just to provide the display part which displays, and the display part which displays an image corresponding to a user's right eye RE. The display device of the present invention may be configured as a head mounted display mounted on a vehicle such as an automobile or an airplane. Further, for example, it may be configured as a head-mounted display built in a body protective device such as a helmet. In this case, the part for positioning the position with respect to the user's body and the part positioned with respect to the part can be used as the mounting portion.

In addition, as an optical system that guides image light to the user's eyes, a configuration in which virtual images are formed by the half mirrors 261 and 281 on a part of the right light guide plate 26 and the left light guide plate 28 is illustrated. The present invention is not limited to this, and an image may be displayed on a display region having an area that occupies the entire or most of the right light guide plate 26 and the left light guide plate 28. In this case, a process of reducing the image may be included in the operation of changing the display position of the image.
Furthermore, the optical element of the present invention is not limited to the right light guide plate 26 and the left light guide plate 28 having the half mirrors 261 and 281, and may be any optical component that allows image light to enter the user's eyes. A diffraction grating, a prism, or a holographic display unit may be used.

  In addition, at least a part of the functional blocks shown in FIGS. 4 and 5 may be realized by hardware, or may be realized by cooperation of hardware and software. As described above, the present invention is not limited to a configuration in which independent hardware resources are arranged. Further, the program executed by the control unit 150 may be stored in the nonvolatile storage unit 121 or another storage device (not shown) in the control device 10. A program stored in an external device may be acquired and executed via the communication unit 117 or the external connector 184. Moreover, the operation part 110 may be formed as a user interface (UI) among the structures formed in the control apparatus 10.

  The processing units of the flowcharts shown in FIGS. 6 and 10 are divided according to main processing contents in order to make the processing of the control unit 150 of the HMD 100 easy to understand. The present invention is not limited by the name. Further, the processing of the control unit 150 can be divided into more processing units according to the processing content, or can be divided so that one processing unit includes more processing. Further, the processing order of the above flowchart is not limited to the illustrated example.

  In addition, when displaying the detection range image 320 in the display area 310, the control unit 150A prevents the display image from interfering with the display image transmissivity and display so that the gestures and hand signs performed by the user are not difficult to see. Adjust the position. The display image includes, for example, a detection range image 320, setting items displayed in the left display area 361 and the right display area 362, a ring image 325, a candidate display area 330, a software keyboard 351, a text input field 352, and the like. .

  DESCRIPTION OF SYMBOLS 1 ... System, 10 ... Control apparatus, 10A ... Case, 11 ... Button, 12, 67 ... LED indicator, 14 ... Track pad, 15 ... Up / down key, 16 ... Changeover switch, 17 ... LED display part, 18 ... Power switch, DESCRIPTION OF SYMBOLS 19 ... Vibrator, 20 ... Image display part (display part), 21 ... Right holding part, 22 ... Right display unit, 23 ... Left holding part, 24 ... Left display unit, 26 ... Right light guide plate, 27 ... Front frame, 28 ... Left light guide plate, 30 ... Headset, 32 ... Right earphone, 34 ... Left earphone, 40 ... Connection cable, 46 ... Connector, 61 ... Camera (detection unit), 63 ... Microphone (voice input unit), 65 ... Illuminance Sensor: 100 ... HMD (display device), 110 ... operation unit, 111 ... 6-axis sensor, 113 ... magnetic sensor, 115 ... GPS receiver, 117 ... communication unit (control) Information output unit), 118 ... memory, 120 ... controller board, 121 ... nonvolatile storage unit, 122 ... storage unit, 123 ... setting data, 124 ... content data, 125 ... pattern registration dictionary, 1251, 1251A, 1251B ... gesture registration Dictionary, 1252, 1252A, 1252B ... Hand sign registration dictionary, 1253, 1253A, 1253B ... External gesture registration dictionary, 1254, 1254A, 1254B ... External hand sign registration dictionary, 126 ... Voice pattern dictionary, 130 ... Power supply unit, 132 ... Battery , 134 ... power supply control circuit, 140 ... main processor, 143 ... operating system, 145 ... image processing unit, 147 ... imaging control unit (detection unit), 149 ... operation control unit, 151 ... detection control unit (operation detection unit), 153 ... Display system 180, audio codec, 182 ... audio interface, 184 ... external connector, 186 ... external memory interface, 188 ... USB connector, 192 ... sensor hub, 194 ... FPGA, 196 ... interface, 200 ... beacon device, 211 ... interface, 213: Receiver, 215 ... EEPROM, 217 ... Temperature sensor, 221 ... OLED unit, 223 ... OLED panel, 225 ... OLED drive circuit, 231 ... Interface, 233 ... Receiver, 235 ... 6-axis sensor, 237 ... Magnetic sensor, 239 ... temperature sensor, 241 ... OLED unit, 243 ... OLED panel, 245 ... OLED drive circuit, 251 ... right optical system, 252 ... left optical system, 261,281 ... half mirror, 300 ... field of view, 310 Display area 320 Detection range image 321 Left detection range image 322 Right detection range image 325 Ring image 326 First detection range image (first detection area) 327 Second detection range image (Second detection area), 330 ... candidate display region, 331 ... left candidate display region, 332 ... right candidate display region, 335 ... candidate character, 335A ... left candidate character, 335B ... right candidate character, 351 ... software keyboard, 352 ... text input field, 361 ... left display area, 362 ... right display area.

Claims (24)

A head-mounted display device mounted on a user's head,
A display unit that displays an image in a display area so as to transmit the outside scene and be visible together with the outside scene;
A detection unit that detects an indicator associated with the user's action in an area including at least a range that passes through the display unit;
An image indicating a detection state in which the detection unit detects the indicator is displayed in the display area, and input content candidates corresponding to at least one of the movement, shape, and position of the indicator detected by the detection unit are displayed. A display control unit to display;
A head-mounted display device comprising:   The head-mounted display device according to claim 1, wherein the display control unit displays an image indicating a detection state of the detection unit in the display area in association with a position where the indicator is visually recognized in the outside scene. .   The display control unit displays the candidate corresponding to at least one of the movement, shape, and position of the indicator detected by the detection unit at a position corresponding to an image indicating a detection state of the detection unit. 2. A head-mounted display device according to 2.   The display control unit displays, in the display area, an image indicating a detection area where the detection unit detects the indicator as an image indicating a detection state of the detection unit, and the detection unit detects the indicator. The head-mounted display device according to claim 1, wherein the candidate corresponding to at least one of movement, shape, and position is displayed at a position corresponding to the image indicating the detection area in the display area.   The display control unit displays the candidate in a candidate display area of the display area, and displays the candidate display area on a position corresponding to the image indicating the detection area or the indicator detected in the detection area. The head-mounted display device according to claim 4, wherein the head-mounted display device is displayed at a corresponding position.   When the detection unit detects the indicator at a position corresponding to the candidate display area, the display control unit is configured to display the candidate display based on at least one of the detected movement, shape, and position of the indicator. The head-mounted display device according to claim 5, wherein one of the plurality of candidates displayed in an area is selected.   When the detection unit detects the indicator at a position corresponding to the candidate character displayed in the display area, the display control unit detects at least one of the detected movement, shape, and position of the indicator. The head-mounted display device according to any one of claims 1 to 6, wherein any one of the plurality of candidate characters displayed in the candidate display area is selected based on the above.   The display control unit displays a plurality of the candidates, and selects the corresponding candidate when the moving direction of the indicator detected by the detection unit corresponds to any direction in which the candidate is displayed. The head-mounted display device according to any one of claims 1 to 7.   9. The display control unit according to claim 1, wherein when the shape of the indicator detected by the detection unit is the same shape for a preset time or longer, the candidate is displayed. Head-mounted display device.   The head mounted according to any one of claims 1 to 9, wherein the display control unit displays the candidate when the position of the indicator detected by the detection unit overlaps a preset position. Type display device.   The head-mounted type according to any one of claims 1 to 10, wherein the display control unit displays the candidate when the indicator detected by the detection unit satisfies a preset condition. Display device.   The display control unit, as an image showing the detection area, an image showing a detection area for detecting a pointer including the left hand of the user and an image showing a detection area for detecting the pointer including the right hand of the user The head-mounted display device according to claim 1, wherein: The detection unit detects a first indicator including the left hand of the user and a second indicator including the right hand of the user as the indicator,
The display control unit displays the first operation image to be operated by the first indicator detected by the detection unit and the second operation image to be operated by the second indicator. The head-mounted display device according to any one of claims 1 to 12, which is displayed in an area.   14. The control information output unit according to claim 1, further comprising a control information output unit configured to output control information to an external device corresponding to at least one of the movement, shape, and position of the indicator detected by the detection unit. Head-mounted display device. A control information output unit that outputs control information to an external device corresponding to at least one of the movement, shape, and position of the indicator detected by the detection unit;
The detection unit detects a first indicator including the left hand of the user and a second indicator including the right hand of the user as the indicator,
The control information output unit includes at least first information corresponding to the operation of the first indicator detected by the detection unit and second information corresponding to the operation of the second indicator detected by the detection unit. The head-mounted display device according to claim 1, wherein the control information including one is output. It has a voice input part to input voice,
The head-mounted display device according to claim 1, wherein the detecting unit detects the indicator when the voice input unit inputs a preset voice. An operation detection unit that detects an operation is provided.
The head-mounted display device according to any one of claims 1 to 16, wherein the detection unit shifts to a state of detecting the indicator when an operation preset by the operation detection unit is detected. A gesture indicating a preset movement trajectory of the user's finger as the indicator, a gesture registration dictionary associating the candidate, a hand sign indicating a preset shape of the finger, A storage unit that stores at least one of the hand sign registration dictionaries associated with the candidates;
The display control unit selects the gesture or the hand sign by matching at least one of the movement, shape, and position of the finger detected by the detection unit with the gesture registration dictionary or the hand sign registration dictionary. The head-mounted display device according to claim 1, wherein the candidate corresponding to the gesture or the hand sign is displayed. Detecting the indicator associated with the action of another user from a direction different from the detection direction of the indicator associated with the action of the user;
A storage unit storing external dictionary information for displaying candidates corresponding to at least one of the movement, shape, and position of the indicator associated with the movement of the other user detected from a different direction; ,
The display control unit identifies and identifies the candidate corresponding to at least one of the movement, shape, and position of the indicator associated with the movement of the other user based on the external dictionary information. The head-mounted display device according to any one of claims 1 to 17, wherein the candidate is displayed. It has a communication unit that communicates with an external head-mounted display device,
The head-mounted display device according to claim 19, wherein the external dictionary information is transmitted to the external head-mounted display device by the communication unit. The detection unit optically detects the other user,
The display control unit optically detects at least one of the movement, shape, and position of the indicator associated with the operation of the other user based on the detection result of the detection unit. An area is displayed in the display area, and at least one of the movement, shape and position of the indicator associated with the user's movement is optically detected in the display area excluding the first detection area. The head-mounted display device according to claim 19 or 20, wherein the second detection area is displayed.   The display control unit identifies the candidate corresponding to at least one of the movement, shape, and position of the indicator detected by the detection unit in the first detection area based on the external dictionary information, The head-mounted display according to claim 21, wherein in the second detection area, the candidate corresponding to at least one of the movement, shape, and position of the indicator detected by the detection unit is specified based on the dictionary information. apparatus. A display unit that displays an image in a display area so that it can be seen through the outside scene and visible together with the outside scene, and an indicator associated with the user's action is detected in an area that includes at least the range that passes through the display section. A program that is executed by a computer mounted on a head-mounted display device that is mounted on the head of the user,
On the computer,
A procedure for displaying an image indicating a detection state in which the detection unit detects the indicator on the display area;
A procedure for displaying input content candidates corresponding to at least one of the movement, shape, and position of the indicator detected by the detection unit;
A program that executes A display unit that displays an image in a display area so as to transmit the outside scene and be visible together with the outside scene, and an indicator associated with the user's action is detected in an area including at least the range that transmits the display section. A method for controlling a head-mounted display device mounted on the user's head, comprising:
Displaying an image indicating a detection state in which the detection unit detects the indicator on the display area;
Displaying input candidate candidates corresponding to at least one of the movement, shape, and position of the indicator detected by the detection unit;
A control method for a head-mounted display device, comprising:

Images