JP2014142751A - Display device, head-mounted type display device, and method for controlling display device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a head-mounted type display device which is capable of providing a user interface capable of executing various kinds of processing with a simple and intuitive operation.SOLUTION: A display device comprises: an indicator detection part detecting a plurality of indicators operated by a user; a pointer display part detecting each indicator by the indicator detection part and displaying a pointer P operated in conjunction with each of the indicators in a virtual image display area 25; and an instruction execution part 55 executing a predetermined instruction corresponding to the movement of a plurality of pointers P, when the plurality of pointers are moved to the vicinity of a boundary 27 between the virtual image display area 25 and an outside view recognition area 26 being an area except the virtual image display area 25, according to the operation of each indicator.

Description

  The present invention relates to a display device that can be operated by an indicator operated by a user, a head-mounted display device, and a display device control method.

  2. Description of the Related Art Conventionally, as a means for operating a head-mounted display device (Head Mounted Display (HMD)), which is a display device mounted on the head, a predetermined motion of a human finger is detected and the motion is handled. What performs processing is known. For example, in Patent Document 1, an input operation corresponding to the movement of a finger is determined by imaging an LED (light emitting body) mounted on the user's finger with a video camera, and a predetermined process is performed based on the determined input operation. Is disclosed.

JP 2000-29619 A

  However, in the case of the above-described prior art, control is complicated because it is necessary to recognize a complicated movement of the user's finger with a video camera. In addition, there is a problem that the types of control are limited or misrecognized easily despite the complicated mechanism. Furthermore, there is a problem that the user has to remember what kind of processing the movement of the finger is to perform, and surely perform the movement, which is not always convenient.

  In view of the above-described problems, the present invention provides a display device, a head-mounted display device, and a display device control method capable of providing a user interface capable of executing various processes with simple and intuitive operations. The purpose is to do.

  The display device of the present invention includes a pointer detection unit that detects a plurality of pointers operated by a user, and a pointer that operates in conjunction with each of the pointers by detecting each pointer with the pointer detection unit. In the vicinity of the boundary between the predetermined display area and the display peripheral area that is an area other than the predetermined display area in accordance with the operation of each indicator. And an instruction execution unit that executes a predetermined instruction in accordance with the movement of a plurality of pointers.

  A head-mounted display device of the present invention is a transmissive head-mounted display device that allows a user to visually recognize an image as a virtual image, and includes an indicator detection unit that detects a plurality of indicators operated by the user, A pointer display unit that detects each indicator by the indicator detection unit and displays a pointer that operates in association with each indicator in a predetermined display area; An instruction execution unit that executes a predetermined instruction according to the movement of a plurality of pointers when the pointer moves to the vicinity of a boundary between the predetermined display area and a display peripheral area that is an area other than the predetermined display area; It is provided with.

  According to the display device control method of the present invention, the display device detects a plurality of indicators operated by the user, detects each indicator, and sets a pointer that operates in conjunction with each indicator. In accordance with the step of displaying in the predetermined display area and the operation of each indicator, the plurality of pointers moved to the vicinity of the boundary between the predetermined display area and the display peripheral area other than the predetermined display area A step of executing a predetermined command in accordance with movement of a plurality of pointers.

  According to these configurations, the user operates each indicator so as to move a plurality of pointers to the vicinity of the boundary of the display area (so as to be positioned in the vicinity of the boundary). Can be executed. That is, since the user can execute a predetermined command by a simple (simple) and intuitive operation without performing a complicated operation, it is easy to use. Further, since it can be performed with a simple operation (no complicated operation is required), it is possible to provide a user interface that operates stably and has few misrecognitions.

  In the display device of the present invention, it is preferable that the instruction execution unit executes different instructions in accordance with the movement order to the vicinity of the boundaries of the plurality of pointers.

  According to this configuration, since various instructions can be assigned according to the movement order of the pointers when moving a plurality of pointers (indicators) to the vicinity of the boundary, the user can respond to various needs of users. An interface can be provided.

  In the display device of the present invention, it is preferable that the instruction execution unit displays a frame image along the inner edge of the display area when at least one of the plurality of pointers moves near the boundary.

  According to this configuration, when the pointer moves to the vicinity of the boundary, a state (frame image) is displayed along the inner edge of the display area, so that there is some state change for the user (for example, A mode in which a predetermined command is received (command reception mode) can be recognized.

  In the display device of the present invention, when the instruction execution unit detects that the plurality of indicators have been operated in a direction to move from the predetermined display region to the display peripheral region across the boundary by the indicator detection unit, It is preferable to execute these instructions.

  In the display device of the present invention, when the instruction execution unit detects that the plurality of indicators are operated in a direction of moving from the display peripheral region to the predetermined display region across the boundary by the indicator detection unit, It is preferable to execute these instructions.

  According to these configurations, when an operation of moving a plurality of indicators from a predetermined display area toward the display peripheral area is performed, or an operation of moving from the display peripheral area toward the predetermined display area is performed. A predetermined command is executed. That is, a predetermined command is executed when an operation in which a plurality of indicators move beyond the boundary is performed. Thereby, it is possible to provide a user interface with high expandability (high flexibility).

  In the display device of the present invention, it is preferable that the command execution unit executes a predetermined command when the pointer detection unit detects operations of the plurality of pointers in which the plurality of pointers move along the boundary.

  In the display device of the present invention, it is preferable that the instruction execution unit executes different instructions according to the moving directions of the plurality of pointers when the plurality of pointers move along the boundary.

  According to these configurations, a predetermined command is executed when an operation of moving a plurality of pointers (indicators) along the boundary is performed. Further, by executing different commands according to the direction in which the plurality of pointers move along the boundary, it is possible to provide a highly expandable (highly flexible) user interface.

  In the display device of the present invention, the instruction execution unit is displayed in a predetermined display area when at least one of the plurality of pointers is rotated while the plurality of pointers are in the vicinity of the boundary. It is preferable to rotate the image in the direction of rotation of the indicator.

  According to this configuration, the displayed image can be rotated by rotating the indicator. For example, all of the plurality of indicators may be rotated in the same direction, or one of the plurality of indicators may be rotated with another indicator as a fulcrum.

  In the display device of the present invention, it is preferable that the instruction execution unit executes a predetermined instruction according to the operation of each indicator in the vicinity of the boundary.

  According to this configuration, it is possible to execute a predetermined command according to the operation content (operation content) of a plurality of indicators near the boundary. As an operation of the indicator, for example, an operation in which two indicators (two pointers) approach in the vicinity of the boundary, or an operation in which the two indicators move in the depth direction of the display area (the user's visual field direction) (switch And the operation of maintaining the state in which the two indicators have moved in the depth direction of the display area (the operation of maintaining the state where the switch is pressed), and the like.

  In the display device of the present invention, when the instruction execution unit detects an operation in which a plurality of indicators sandwich an icon displayed in a predetermined display region across the boundary from the display peripheral region by the indicator detection unit, Preferably, the assigned predetermined instruction is executed.

  According to this configuration, it is possible to execute an instruction (for example, execution of an application) assigned to an icon when an operation of grasping the icon with a plurality of indicators is performed, and it is intuitive and has good operability. Can provide a user interface.

  In the display device of the present invention, it is preferable that the command execution unit executes a predetermined command corresponding to the displayed content.

  According to this configuration, for each content to be displayed, a command suitable for the content can be executed. Note that the content means information content including an image (still image, moving image), sound, or the like, an application, or the like.

  The display device described above, wherein the display device is a transmissive display device that allows a user to visually recognize an image as a virtual image, and the indicator detection unit is configured so that the user can use the transmissive display device on the user side. A plurality of indicators operated on the outer front side of the display device on the opposite side of the display device are detected.

  According to this configuration, in the transmissive display device, it is possible to detect the indicator operated outside (front) beyond the display device as viewed from the user.

  In the display device described above, the display device is a transmissive display device that allows a user to visually recognize an image as a virtual image, and the image display unit of the transmissive display device uses image data to be displayed. An image light generation unit that generates and emits image light that represents the image, and an optical member that guides the emitted image light to the user's eyes, and transmits the light of the outside scene through the optical member to generate the image light. And entering the user's eyes.

  According to this configuration, the present invention can also be applied to a transmissive display device in which a user can view a virtual image and an outside scene at the same time.

  In the head-mounted display device of the present invention, it is preferable that the indicator detection unit detects a plurality of indicators by imaging the front of the user with the head-mounted display device mounted.

  According to this configuration, the indicator operated by the user is detected based on the imaging result in front of the user when the head-mounted display device is mounted. In other words, the user can wear the head-mounted display device and image and detect the indicator operated in front of the eyes (in front of the device).

It is explanatory drawing which shows the external appearance structure of the head mounted display apparatus which concerns on 1st Embodiment. It is a figure explaining an example of a virtual image and an outside scene which a user can recognize visually. It is a block diagram explaining the functional structure of the head mounted display apparatus which concerns on 1st Embodiment. It is a figure explaining the operation procedure which displays a menu screen. It is a figure explaining the operation procedure which performs a menu item. It is a figure explaining the operation procedure which performs a menu item. It is a figure explaining the process sequence which displays another menu screen. It is a figure explaining the operation procedure which performs a copy process. It is a figure explaining the operation procedure which performs an expansion process. It is a figure explaining the operation procedure which performs a reduction process. It is a figure explaining the other operation procedure which performs an expansion process. It is a figure explaining the other operation procedure which performs a reduction process. It is a figure explaining the operation procedure which performs a rotation process. It is a figure explaining the operation procedure which performs an enlargement process using the palm. It is explanatory drawing which shows the external appearance structure of the head mounted display apparatus which concerns on 2nd Embodiment. It is a block diagram explaining the functional structure of the head mounted display apparatus which concerns on 2nd Embodiment.

[First Embodiment]
Hereinafter, a display device, a head-mounted display device, and a display device control method according to an embodiment of the present invention will be described with reference to the accompanying drawings. FIG. 1 is an explanatory diagram showing an external configuration of a head-mounted display device HM as a display device according to an embodiment of the present invention. The head-mounted display device HM is a display device that a user wears on the head, and is also referred to as a head mounted display (HMD). The head-mounted display device HM of this embodiment is an optically transmissive head-mounted display device HM (a transmissive head-mounted display device, a see-through type) that allows a user to visually recognize a virtual image and at the same time directly view an outside scene. -A head-mounted display) having a spectacle shape.

  The head-mounted display device HM includes a right holding unit 11, a right display driving unit 12, a left holding unit 13, a left display driving unit 14, a right optical image display unit 15, and a left optical image display unit 16. , Sensor 17.

  The right optical image display unit 15 and the left optical image display unit 16 are disposed at positions corresponding to the right and left eyes of the user when the head-mounted display device HM is mounted. One end of the right optical image display unit 15 and one end of the left optical image display unit 16 are connected at positions corresponding to the user's eyebrows when the head-mounted display device HM is mounted. The right holding unit 11 extends from the end ER that is the other end of the right optical image display unit 15. Similarly, the left holding unit 13 extends from the end EL which is the other end of the left optical image display unit 16.

  The right holding unit 11 is substantially perpendicular to the right optical image display unit 15 from the end ER of the right optical image display unit 15 to a position corresponding to the user's temporal region when the head-mounted display device HM is mounted. It is a member provided by stretching so as to be formed. Similarly, the left holding unit 13 is connected to the left optical image display unit 16 from the end EL of the left optical image display unit 16 to a position corresponding to the user's temporal region when the head-mounted display device HM is worn. It is a member provided by extending so as to form a substantially right angle. The right holding unit 11 and the left holding unit 13 hold the head-mounted display device HM on the user's head like a temple of glasses.

  The right display drive unit 12 is located inside the right holding unit 11, in other words, on the side facing the user's head when the head-mounted display device HM is worn, and the end of the right optical image display unit 15. It is arranged on the part ER side. The left display driving unit 14 is disposed inside the left holding unit 13 and on the end EL side of the left optical image display unit 16.

  The sensor 17 is an imaging device arranged at a position corresponding to the user's eyebrow when the head-mounted display device HM is worn. The sensor 17 images an outside scene in the user's viewing direction (in front of the user) in a state where the head-mounted display device HM is worn, and a plurality of indicators (two instructions in the present embodiment) operated by the user. The image including the body is acquired, and an image signal of the acquired image is output. That is, the indicator operated on the side opposite to the user side in the head-mounted display device HM (outside front of the head-mounted display device HM) is imaged. The imaging range by the sensor 17 of this embodiment is wider than the user's field of view 21 (see FIG. 2) when the head-mounted display device HM is worn.

  As the indicator, for example, a user's fingertip or hand, a ring-type input device worn by the user, a pen tip of the user, a light emitter of a remote controller, or the like can be used. When the user's fingertip or pen tip is used as the indicator, for example, the finger tip or pen tip may be recognized as a feature point (specific point) and the position may be detected. When the user's hand is used as the indicator, the palm curve may be recognized as a feature point and the position may be detected. When a ring type input device is used, for example, an infrared light emitter (LED or the like) may be attached to the ring type input device, and the position may be detected by the infrared light, or the ring type input device is attached. The user's fingertip may be recognized as a feature point to detect the position. Furthermore, an arbitrary object virtually held by the user can be used as the indicator. “Anything that is virtually held” means, for example, a pen that is virtually assumed to follow the hand of the user when the shape of the user's hand becomes a shape of holding a pen. Even in such a case, similarly to the above, an arbitrary place of a virtually held object (for example, a virtually assumed pen nib) is recognized as an indicator, and the pen tip The locus or the like can be recognized as a feature point, and the position can be detected.

  In this embodiment, one sensor 17 is illustrated, but a plurality of sensors 17 may be provided. In the present embodiment, an imaging device (imaging sensor) is illustrated as the sensor 17, but the present invention is not limited to this, and the type (detection method) is not limited as long as the indicator can be detected. For example, a laser sensor, an ultrasonic sensor, or the like can be used as the sensor 17, and a plurality of types of detection methods may be combined.

  The head-mounted display device HM further includes a right earphone 18 for the right ear and a left earphone 19 for the left ear. The right earphone 18 and the left earphone 19 are respectively attached to the right and left ears when the user wears the head-mounted display device HM.

  Next, an example of a virtual image and an outside scene that can be visually recognized by the user wearing the head-mounted display device HM will be described with reference to FIG. As shown in FIG. 2, a virtual image 22 generated by the head-mounted display device HM is displayed in the field of view 21 of the user wearing the head-mounted display device HM. Further, the user can see the outside scene 23 through the right optical image display unit 15 and the left optical image display unit 16 in a region other than the region where the virtual image 22 is displayed in the user's field of view 21. (Hereinafter, an area where the virtual image 22 is displayed is referred to as a “virtual image display area 25”, and an area other than this is referred to as an “outside view visually recognizing area 26”). Further, in the head-mounted display device HM of the present embodiment, the outside scene 23 can be seen through the virtual image 22 in the area (virtual image display area 25) in which the virtual image 22 is displayed in the user's field of view 21. It is like that. Although details will be described later, various commands for executing predetermined processing are allocated in the vicinity of the boundary 27 (the edge of the virtual image display area 25) between the virtual image display area 25 and the outside scene visual recognition area 26. It can be executed by operating the two indicators by the user.

  Next, the functional configuration of the head-mounted display device HM will be described with reference to FIG. As shown in FIG. 3, the head-mounted display device HM includes an image display unit 40 that allows a user to visually recognize a virtual image, a sensor 17, a right earphone 18, a left earphone 19, a power supply unit 71, and an interface 72. And a control unit 30 that controls these units.

  The power supply unit 71 supplies power to each unit of the head-mounted display device HM. The interface 72 is for connecting various external devices OA that are sources of contents such as images (still images, moving images) and audio. Examples of the external device OA include a personal computer (PC), a mobile terminal (such as a mobile phone and a smartphone), and a game terminal. As the interface 72, for example, a USB interface, a micro USB interface, a memory card interface, a wireless LAN interface, or the like can be employed.

  The control unit 30 includes a CPU (Central Processing Unit) 50 and a storage unit 60. The storage unit 60 includes a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and stores various computer programs. The CPU 50 reads out and executes a computer program from the storage unit 60, thereby operating as an operating system (OS) 51, an image processing unit 52, a display control unit 53, an image analysis unit 54, a command execution unit 55, and a voice processing unit 56. Function.

  The image processing unit 52 generates a signal to be supplied to the image display unit 40 based on content input through the interface 72 and transmits the signal to the image display unit 40. The signal supplied to the image display unit 40 differs between the analog format and the digital format.

  For example, in the case of an analog format, the image processing unit 52 generates and transmits a clock signal PCLK, a vertical synchronization signal VSync, a horizontal synchronization signal HSync, and image data Data. Specifically, the image processing unit 52 acquires an image signal (for example, an analog signal composed of 30 frame images per second) included in the content, and from the acquired image signal, a vertical synchronization signal VSync, A synchronization signal such as the horizontal synchronization signal HSync is separated. Then, the image processing unit 52 generates a clock signal PCLK using a PLL circuit (not shown) or the like according to the separated vertical synchronization signal VSync, horizontal synchronization signal HSync, and the like. Further, the image processing unit 52 converts the analog image signal from which the synchronization signal has been separated into a digital image signal using an A / D conversion circuit (not shown) or the like to generate image data Data.

  On the other hand, in the case of the digital format, the image processing unit 52 generates and transmits the clock signal PCLK and the image data Data. Specifically, when the content is in a digital format, the clock signal PCLK is output in synchronization with the image signal. For this reason, A / D conversion of the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the analog image signal becomes unnecessary.

  Note that the image processing unit 52 may execute image processing such as resolution conversion processing, color tone correction processing such as luminance / saturation adjustment, and keystone correction processing on the image data Data as necessary.

  The display control unit 53 controls the image display state in the image display unit 40 by generating control signals for controlling the right display drive unit 12 and the left display drive unit 14 and transmitting them to the image display unit 40. Specifically, the display control unit 53 uses the control signal to turn on / off the drive of the right LCD 84 by the right LCD control unit 83, drive the ON / OFF of the right backlight 82 by the right backlight control unit 81, and the left LCD. The right display drive unit 12 and the left display drive unit 14 are individually controlled by controlling ON / OFF of the left LCD 94 by the control unit 93 and ON / OFF of the left backlight 92 by the left backlight control unit 91, respectively. The generation and emission of image light by each of the above are controlled. For example, the display control unit 53 causes both the right display driving unit 12 and the left display driving unit 14 to generate image light, generates only one image light, or does not both generate image light.

  The image analysis unit 54 acquires an image signal of an image captured by the sensor 17 and analyzes the acquired image signal. The image analysis unit 54 detects two indicators by image analysis and outputs position information of each indicator. The image processing unit 52 and the display control unit 53 cause the image display unit 40 to display a pointer P (see FIG. 4A) corresponding to each indicator based on the output positional information of the indicator. In addition, the indicator detection part in a claim has the sensor 17 and the image analysis part 54 as main components. Further, the pointer display unit in the claims has the image processing unit 52 and the display control unit 53 as main components.

  The command execution unit 55 executes various commands according to the movement position and movement direction of each pointer P accompanying the operation of the two indicators. Specifically, when the two pointers P are moved to the vicinity of the boundary 27 between the virtual image display area 25 and the outside scene visual recognition area 26 (both refer to FIG. 2), or the two pointers move from the virtual image display area 25 to the boundary 27. When the two pointers are operated to move from the outside scene viewing area 26 to the virtual image display area 25 beyond the boundary 27, the two pointers P are operated. Is moved along the boundary 27, when an operation to sandwich an icon displayed in the virtual image display area 25 by two indicators (two pointers P) (operation to grab the icon) is performed, Alternatively, when a pointer is operated near the boundary in the virtual image display area 25, various commands (processing) assigned in advance are executed (details will be described later). That is, in the present embodiment, “predetermined display area” in the claims corresponds to the virtual image display area 25, and “display peripheral area” corresponds to the outside scene visual recognition area 26. Further, the boundary 27 between the virtual image display area 25 and the outside scene visually recognizing area 26 indicates a limit at which the pointer P can be displayed (the pointer P can be moved).

  The audio processing unit 56 acquires an audio signal included in the content, amplifies the acquired audio signal, and transmits the amplified audio signal to the right earphone 18 and the left earphone 19.

  The image display unit 40 includes a right display drive unit 12, a left display drive unit 14, a right light guide plate 86 as the right optical image display unit 15, and a left light guide plate 96 as the left optical image display unit 16. doing. In the following description, the right optical image display unit 15 and the right light guide plate 86, and the left optical image display unit 16 and the left light guide plate 96 are also collectively referred to as “optical members”.

  A right display drive unit 12, a right backlight (BL) control unit 81 and a right backlight (BL) 82 functioning as a light source, a right LCD control unit 83 and a right LCD 84 functioning as display elements, and a right projection optical system 85 Contains. In the following description, the right backlight control unit 81, the right backlight 82, the right LCD control unit 83, and the right LCD 84 are also collectively referred to as “image light generation unit”.

  The right backlight control unit 81 drives and controls the right backlight 82 based on the input control signal. The right backlight 82 is a light emitter such as an LED or electroluminescence (EL). The right LCD control unit 83 drives and controls the right LCD 84 based on the input clock signal PCLK, vertical synchronization signal VSync, horizontal synchronization signal HSync, and image data Data for the right eye. The right LCD 84 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix.

  The right projection optical system 85 projects image light emitted from the right LCD 84, and is configured using, for example, a collimator lens. The right light guide plate 86 as the right optical image display unit 15 is formed of a light-transmitting resin material or the like, and reflects the image light output from the right projection optical system 85 along a predetermined optical path while being reflected by the user. Lead to the right eye. The right projection optical system 85 and the right light guide plate 86 are collectively referred to as “light guide unit”. The light guide unit can use any method as long as it forms a virtual image in front of the user's eyes using image light. For example, a diffraction grating (hologram element as an example) may be used, or transflective reflection may be used. A film may be used.

  As with the right display drive unit 12, the left display drive unit 14 includes a left backlight (BL) control unit 91, a left backlight (BL) 92, a left LCD control unit 93, a left LCD 94, and left projection optics. (Like the right display drive unit 12, the left backlight control unit 91, the left backlight 92, the left LCD control unit 93, and the left LCD 94 are collectively referred to as an “image light generation unit”. Call). Since the configuration and function of each element included in the left display driving unit 14 are the same as those included in the right display driving unit 12, the description thereof is omitted here. The left light guide plate 96 as the left optical image display unit 16 is formed of a light transmissive resin material or the like, and is used while reflecting the image light output from the left projection optical system 95 along a predetermined optical path. To the left eye.

  In this way, the image light guided to both eyes of the user of the head-mounted display device HM forms an image on the retina, so that the user can enter the virtual image display area 25 as shown in FIG. The virtual image 22 is visually recognized.

  Next, referring to FIGS. 4 to 14, various commands are issued by the operation of the pointer P displayed on the virtual image display area 25 by the head-mounted display device HM in accordance with the operation of the two indicators by the user. An example of the procedure to be executed will be described. In the following description, a case will be described in which two fingers of a user (for example, the index fingers (fingertips of both hands) or the thumb and index finger (fingertips) of one hand) are detected as indicators.

  FIGS. 4 to 6 show menu screen display processing and menu item execution procedures (menu screen display and user interface for executing menu items) by the user's operation of two fingers (operation of two pointers). Is a diagram for explaining an example.

  As shown in FIG. 4A, when the user's right hand index finger and left hand index finger (two indicators) are detected in the virtual image display area 25 (more precisely, two fingers are displayed in the virtual image display area 25). The head-mounted display device HM detects a circular pointer P (pointers P1 and P2) in the virtual image display area 25 based on the detection position of the two fingers. Is displayed. Further, the head-mounted display device HM can execute a command assigned in the vicinity of the boundary 27 between the virtual image display area 25 and the outside scene visual recognition area 26 (hereinafter referred to as “input standby mode”). Execute the process to transit to.

  When the indicator (user's finger) is detected outside the virtual image display area 25, the head-mounted display device HM displays, for example, an arrow-shaped pointer indicating the detection direction of the indicator, or The user is notified that the indicator is outside the virtual image display area 25 by, for example, displaying a pointer with a dotted line. In this case, the head-mounted display device HM does not transition to the “input standby mode”.

  Next, as shown in FIG. 4B, in the state of the input standby mode, the pointer P1 is located near the boundary 27 by the operation of the right index finger (in FIG. 4B, as an example, the right side of the virtual image display area 25 The head-mounted display device HM moves to the vicinity of the boundary 27 (near the inside of the virtual image display area 25 at the boundary 27 between the virtual image display area 25 and the outside scene visual recognition area 26). Execute processing to transition to a mode that accepts the assigned command (hereinafter referred to as “command acceptance mode”) (execute a transition command to the command acceptance mode), and change the shape of the pointer P1 into a semicircle indicate. At the same time, a frame 101 (a frame image, a shaded portion in the drawing) is displayed along the boundary 27 (along the virtual image display area 25 of the boundary 27). By displaying the deformation of the shape of the pointer P1 and the frame 101, the user can recognize that the command acceptance mode has been entered. When the semicircular pointer P1 is moved to a position away from the vicinity of the boundary 27 of the virtual image display area 25 (when moved toward the center of the virtual image display area 25), the shape thereof is the original circular shape. Return to.

  Next, in the instruction acceptance mode, the pointer P2 moves to the vicinity of the boundary 27 (in FIG. 4B, as an example, the vicinity of the lower boundary 27d of the virtual image display area 25) by operating the left index finger. Then, the head-mounted display device HM executes a command for displaying a menu screen assigned in the vicinity of the boundary 27d (hereinafter referred to as “menu screen display command”), and is shown in FIG. A menu screen D01 composed of various icons IC (menu items) is displayed in the virtual image display area 25. This menu screen display command is executed when both the pointers P1 and P2 (two indicators) are located in the vicinity of the boundary 27.

  The user touches various icon ICs on the displayed menu screen D01 to execute a function assigned to the icon IC. In addition to touching the icon IC, a function assigned to the icon IC can also be executed when an operation for gripping the icon IC (an operation for pinching) is performed.

  For example, as shown in FIG. 5, an operation is performed in which the user grasps the icon IC in the lower right corner of the virtual image display area 25 by moving the thumb and forefinger so as to straddle the boundary 27 a and the boundary 27 d from the outside scene viewing area 26 side. If performed (when performing an operation such as holding the icon IC with the pointers P1 and P2), an instruction assigned to the icon IC (here, an activation instruction for the software keyboard KB) can be executed (hereinafter referred to as this Such an operation is called “fingertip action”). That is, a command for executing the fingertip action is assigned to the lower right corner of the virtual image display area 25. In FIG. 5, the fingertip action is performed on the icon IC in the lower right corner of the virtual image display area 25, but the icon IC in the lower left corner is operated by the same operation (with the movement of two fingers (indicator)). It is possible to execute a fingertip action on the screen. Further, when performing a fingertip action on the icon IC in the upper right corner (or upper left corner) of the virtual image display area 25, it is executed by moving two fingers (indicators) as shown in FIG. Can do.

  Note that the arrangement of the icon ICs on the menu screen D01 in FIG. 4C is merely an example, and the present invention is not limited to this. 4C illustrates the menu screen D01 in which the icon IC is arranged on the left end side and the corner of the virtual image display area 25. For example, the icon IC is arranged on the right end side of the virtual image display area 25. A menu screen may be displayed, or a menu screen in which icon ICs are arranged at both left and right ends may be displayed. In other words, the icon IC may be arranged on the inner peripheral portion (edge) of the virtual image display area 25 so as not to obstruct the field of view as much as possible. Alternatively, a menu screen in which the icon IC is arranged near the center (near the center) of the virtual image display area 25 may be displayed. Further, the content of the menu screen to be displayed and the arrangement of the icon ICs may be changed according to the movement order of the pointer P to the vicinity of the boundary 27.

  In FIG. 4C, only the menu screen D01 including the icon IC is displayed. For example, other virtual images (for example, contents such as images and videos) are displayed behind the menu screen D01. (The menu screen D01 may be displayed so as to be superimposed on another virtual image).

  In the above description, the process of transitioning to the “command acceptance mode” is performed by moving the pointer P (pointer P1) to the vicinity of the right boundary 27a of the virtual image display area 25. However, this is merely an example. It is not limited to. For example, when the pointer P moves to the vicinity of the left boundary 27b, the vicinity of the upper boundary 27c, or the vicinity of the lower boundary 27d, it is possible to execute the process of transitioning to the “command acceptance mode”. is there. Further, when changing to the command acceptance mode, the thickness and color of the frame 101, the shape and size of the pointer P after moving to the vicinity of the boundary 27, and the like are changed depending on which pointer P has moved to the vicinity of the boundary 27. You may do it.

  In the above description, it is assumed that the “menu screen display command” is assigned to the vicinity of the lower boundary 27d of the virtual image display area 25. However, this is merely an example, for example, the vicinity of the right boundary 27a, It is also possible to assign an instruction to the vicinity of the boundary 27b, the vicinity of the upper boundary 27c, or the like (that is, the position to which the instruction is allocated is not limited as long as it is in the vicinity of the boundary 27). If no instruction is assigned in the vicinity of the boundary 27, the shape of the pointer P may be displayed as a prohibition mark to notify the user of that fact.

  In the figure, for convenience, the frame 101 is shaded, but actually, the frame 101 may be displayed in a color that can be identified by the user (for example, white, gray, red, etc.). Further, the frame 101 may be displayed as a dotted line instead of a solid line, or may be without a frame line. In addition, the circular pointer P1 is displayed in a semicircular shape when the pointer P is moved to the vicinity of the boundary 27 during the transition to the “command reception mode” or during the command reception mode, but this is not limited thereto. It is not a thing. For example, the size of the pointer P may be increased, or a double circular pointer or the like may be displayed. The same applies to the following description.

  Next, with reference to FIG. 7 and FIG. 8, other forms of menu screen display processing and menu item execution by a user's operation of two fingers (operation of two pointers) will be described. Note that the procedure up to the transition to the command reception mode is the same as described above, and thus the description thereof is omitted. The same applies to the following description.

  As shown in FIG. 7A, when the pointer P2 is moved to the vicinity of the boundary 27d by the operation of the left index finger in the state of transition to the command reception mode, the head-mounted display device HM is assigned to the vicinity of the boundary 27d. As shown in FIG. 7B, the shape of the pointer P2 is transformed into a semicircular shape and displayed along the boundary 27 (along the virtual image display area 25 side of the frame 101). ), A band-like operation area 102 is displayed. Various commands are assigned to the operation area 102 in accordance with the position.

  Next, as shown in FIG. 7C, with the operation area 102 displayed, the pointers P1 and P2 are moved to the upper right corner of the operation area 102 by, for example, the operation of the thumb and index finger of the left hand. The menu screen D02 is displayed by performing the operation of bringing the finger close (the operation of bringing the pointer P1 and the pointer P2 close). That is, a command (menu screen display command) executed by bringing the two pointers P close to each other is assigned to the upper right corner in the operation area 102.

  In this menu screen D02, icon ICs such as “Copy”, “Cut”, “Paste”, “Home”, “Back”, etc. are displayed as menu items. Is displayed. “Copy”, “Cut”, and “Paste” icons are for copying, cutting, and pasting images and the like. The “Home” icon switches the display image in the virtual image display area 25 to a standard home application image (not shown). The “Back” icon is used to return to the previous operation. The displayed menu item (icon IC) and its display position are merely examples, and the present invention is not limited to this.

  The user performs a function assigned to the icon IC by touching the icon IC on the displayed menu screen D02. Here, as a specific example, a procedure for executing image copying will be described. First, as shown in FIG. 8 (a), the user displays the image G1 to be copied in the virtual image display area 25, and then selects "Copy" with the right index finger (pointer P1) to execute the copy function. To enable. Then, the image G1 as the copy source is designated with the left index finger (pointer P2). Next, as shown in FIG. 8B, the user designates the position to be the copy destination with the left index finger (pointer P2), and selects “Paste” with the right index finger (pointer P1). To enable the paste function. As a result, as shown in FIG. 8C, the image G1 designated as the copy source is copied to the position designated as the copy destination. In the above description, the copy operation is performed using both hands. However, the same operation can be performed using only one hand (for example, the right index finger).

  In the above description, an example in which a menu screen display command is assigned to the upper right corner of the operation area 102 has been described. However, various instructions can be assigned to predetermined positions in the operation area 102. For example, a click function and a drag function similar to those of a mouse can be assigned to the upper left corner of the operation area 102. In this case, by moving two fingers (two indicators) to the upper left corner of the operation area 102, both fingers are moved in the depth direction of the virtual image display area 25 (the user's view direction). A click function is realized (by pressing the switch), and both fingers are moved in the depth direction of the virtual image display area 25 (the switch is pressed). By doing so, the drag function can be realized.

  Next, an example of a procedure (user interface for executing the enlargement / reduction process) in the case of executing the image enlargement / reduction process (enlargement / reduction instruction) will be described with reference to FIGS.

  As shown in FIG. 9 (a), after the user has designated the range 111 to be enlarged in the image G2 in the state of transition to the command acceptance mode, the right index finger is moved from the virtual image display area 25 to the boundary 27a. An operation of moving the left index finger from the virtual image display area 25 across the boundary 27d to the outer scene visual recognition area 26 diagonally to the lower left (arrow B direction) is performed obliquely in the upper right direction (in the direction of arrow A). In other words (in other words, the pointer P1 moves diagonally to the upper right with respect to the boundary 27a, and the pointer P2 moves diagonally to the lower left with respect to the boundary 27d, and moves from the virtual image display area 25 to the outside scene viewing area 26 across the boundaries 27a and 27d. The head-mounted display device HM enlarges the image in the designated range 111 at a predetermined magnification, as shown in FIG. 9B. It performs much processing (enlargement instruction), and displays the enlarged image 112 which is the execution result.

  At this time, the head-mounted display device HM changes the shape of the pointers P1 and P2 to a graphic indicating that the enlargement process has been performed, and the enlargement process has been performed at a predetermined position in the virtual image display area 25. A plus sign 113 is displayed. Further, when the user reaches the limit of the enlargement process by repeating the enlargement operation (when the enlargement cannot be performed any more), the head-mounted display device HM changes the pointer to the shape of the prohibition mark. And display.

  As shown in FIG. 9 (c), the right index finger is moved upward (in the direction of arrow C) along the boundary 27a, and the left hand index finger is moved in the left direction (in the direction of arrow D) along the boundary 27d. (In other words, when the pointer P1 is operated to move upward along the boundary 27a and the pointer P2 is moved leftward along the boundary 27d), the enlargement process is performed as described above. (Enlargement instruction) can be executed.

  On the other hand, as shown in FIG. 10 (a), after the user has designated the range 111 to be reduced in the image G1 in the state of transition to the command acceptance mode, the right index finger is moved from the outside view viewing area 26 to the boundary 27a. The operation of moving the left index finger to the virtual image display area 25 diagonally in the lower left direction (direction of arrow E) and moving the left index finger from the outside scene viewing area 26 to the virtual image display area 25 diagonally in the upper right direction (direction of arrow F). When performed (in other words, when the pointer P1 and the pointer P2 are operated so as to move from the boundary 27 side to the inside of the virtual image display area 25), the head-mounted display device HM is shown in FIG. As shown, a reduction process (reduction command) for reducing an image in a designated range 111 at a predetermined magnification is executed, and a reduced image 114 as a result of the execution is displayed.

  At this time, the head-mounted display device HM changes the shape of the pointers P1 and P2 to a graphic indicating that the reduction process has been performed, and a minus sign indicating the reduction process at a predetermined position in the virtual image display area 25. 115 is displayed. In addition, when the user reaches the limit of the reduction process by repeating the reduction operation (when the user cannot reduce any more), the head-mounted display device HM changes the pointer to the shape of the prohibition mark. Change and display.

  As shown in FIG. 10C, the right index finger is moved downward (in the direction of arrow G) along the boundary 27a, and the left hand index finger is moved in the right direction (in the direction of arrow H) along the boundary 27d. (In other words, when the pointer P1 is operated to move downward along the boundary 27a and the pointer P2 is moved right along the boundary 27d), the reduction processing is performed in the same manner as described above. (Reduction instruction) can be executed.

  In the above description, the enlargement process and the reduction process are performed on the range 111 designated by the user. However, for example, the entire image G2 can be enlarged or reduced without specifying the range. In the above description, in order to notify the user that the enlargement process or the reduction process has been performed, the shape of the pointers P1 and P2 is changed to a shape corresponding to the content of the executed process. It is an example. The pointers P1 and P2 may be changed to shapes (graphics) different from the above. Further, the shape of only one of the pointers P1 and P2 may be changed. Alternatively, the shape of both pointers P need not be changed.

  In the above description, the enlargement process or the reduction process of the image G2 is performed by enlarging or reducing the image by a predetermined magnification by one operation. However, the present invention is not limited to this. For example, two fingers are moved. The enlargement ratio and the reduction ratio may be varied according to the direction (angle with respect to the boundary 27), the distance to be moved, the speed to be moved, the timing to be moved, and the like.

  In the above description, in order to notify the user that the enlargement process or the reduction process has been performed, the plus sign 113 indicating the enlargement process and the minus sign 115 indicating the reduction process are displayed. However, the present invention is not limited to this. Absent. For example, an indicator that can recognize the enlargement process or the reduction process may be displayed. That is, as long as it is possible for the user to recognize the processing contents, the shape and means are not limited. The same applies to the following description.

  In the above description, the enlargement / reduction of the image G2 has been described. For example, when text input is performed using a word processor or the like, the text input screen may be enlarged or reduced by the same operation as described above. Is possible.

  In the above description, the procedure for executing the enlargement process and the reduction process using both hands has been described. However, the enlargement process and the reduction process can be performed with one hand (for example, the thumb and index finger of the right hand). For example, as shown in FIG. 11, when the enlargement process is performed with one hand, the thumb and forefinger from the state in which the thumb and forefinger of the right hand are brought close to each other in the virtual image display area 25 (from the state in which the pointer P1 and the pointer P2 are brought closer). Can be performed when an operation is performed to expand the boundary in the directions of the arrow I and the arrow J toward the boundary 27d and the boundary 27a.

  On the other hand, when the reduction process is performed with one hand, as shown in FIG. 12, the thumb and the index finger are moved from the outside scene viewing area 26 to the virtual image display area 25 in the directions of the arrows K and L across the boundaries 27d and 27a. The reduction process can be executed when an operation (in other words, an operation of grasping (squeezing) the corner portion formed by the boundary 27a and the boundary 27d from the outside scene visual recognition area 26 side) is performed.

  In the above description, the enlargement process and the reduction process are performed by operating the two indicators (two pointers P1 and P2) so as to straddle the boundary 27a and the boundary 27d (or move along the boundary 27d). However, this is only an example, and the present invention is not limited to this. Even when the same operation is performed on other boundaries 27 (for example, the boundaries 27b and 27c), the enlargement process and the reduction process can be performed.

  Next, with reference to FIG. 13, an example of a procedure (user interface for executing rotation processing) when executing image rotation processing (rotation command) will be described.

  As shown in FIG. 13A, the image G3 is displayed in the virtual image display area 25, and the right thumb and index finger are rotated counterclockwise so as to straddle the boundary 27b and the boundary 27c in the state of transition to the command reception mode. When the operation is performed (in other words, when the entire right hand is rotated and the pointer P1 and the pointer P2 are rotated counterclockwise), the head-mounted display device HM As shown in FIG. 13B, a process (rotation command) for rotating the displayed image G3 by 90 degrees counterclockwise is executed. At this time, the head-mounted display device HM changes the shape of the pointers P1 and P2 to an arrow-shaped graphic (an arrow-shaped graphic indicating the rotation direction) indicating that the rotation processing has been performed, and displays it.

  In addition, as shown in FIG. 13C, there is a case where an operation is performed in which only the index finger is rotated counterclockwise with the thumb of the right hand as a fulcrum (in other words, only the pointer P2 is rotated counterclockwise). The rotation processing (rotation command) can be executed in the same manner as described above. In the above description, the procedure for rotating the image G3 counterclockwise has been described. However, the image G3 can be rotated 90 degrees clockwise by rotating the indicator (thumb or index finger) clockwise. . It is also possible to perform the same rotation operation using both hands (for example, the index fingers of both hands).

  In the above description, the image G3 is rotated 90 degrees clockwise by one rotation operation. However, this is only an example, and the rotation angle is arbitrary. For example, you may make it rotate 45 degree | times by one rotation operation. Further, the rotation angle may be determined based on the degree of rotation of the finger, the rotation speed, or the like. Alternatively, the rotation angle may be determined based on the number of finger rotations at the same position.

  In the above description, in order to notify the user that the rotation process has been performed, the shape of the pointers P1 and P2 is changed to an arrow-shaped graphic indicating the rotation direction, but this is merely an example. . The pointers P1 and P2 may be changed to shapes (graphics) different from the above. Further, the shape of only one of the pointers P1 and P2 may be changed. Alternatively, the shape of both pointers P need not be changed.

  In the above description, the rotation process is executed by rotating the two indicators (two pointers P1 and P2) in the vicinity of the boundary 27b and the boundary 27c. is not. Even when the same operation is performed on the other boundary 27 (for example, the boundary 27a and the boundary 27d), the rotation process can be executed.

  As described above, according to the first embodiment, each indicator is moved so that the user moves the two pointers P in the vicinity of the boundary 27 of the virtual image display region 25 (so as to be positioned in the vicinity of the boundary 27). A predetermined command (processing) preliminarily assigned in the vicinity of the boundary 27 can be executed simply by operating. That is, since the user can execute a predetermined command by a simple (simple) and intuitive operation without performing a complicated operation, it is easy to use. Further, since it can be performed with a simple operation (no complicated operation is required), it is possible to provide a user interface that operates stably and has few misrecognitions.

  Note that the operation method of the indicator for executing the various commands described in this embodiment (the movement of the pointer P for executing the various commands) is an example, and is not limited thereto.

  Moreover, the allocation positions of various instructions in this embodiment are merely examples, and the present invention is not limited to this. For example, when two indicators are operated to move from the virtual image display area 25 to the outside scene visual recognition area 26 beyond the left and right boundaries 27b and 27a (or the upper and lower boundaries 27c and 27d), the enlargement process is performed. An instruction may be assigned to execute the reduction process when the viewing area 26 is operated to move to the virtual image display area 25 beyond the left and right boundaries 27b and 27a (or the upper and lower boundaries 27c and 27d).

  Further, for example, the enlargement process and the rotation process can be continuously executed by combining and assigning instructions for performing the enlargement / reduction process and the rotation process described in the present embodiment.

  It is also possible to assign instructions other than the instructions described in this embodiment. For example, when a movie is displayed in the virtual image display area 25, a command corresponding to the content is assigned, such as a command for adjusting the volume by performing the same operation as the above-described enlargement / reduction processing. It is possible. In other words, different commands may be assigned to the vicinity of each boundary 27 according to the content.

  In the present embodiment, two fingers (fingertips) are illustrated as two indicators, but the palms of both hands (the palm curve) may be detected as the indicators. For example, when image enlargement processing is performed using the palms of both hands, as shown in FIG. 14, the palms of both hands are detected as indicators and pointers P1 and P2 are displayed, and the palms of both hands (pointers P1 and P2). Is expanded in the direction from the virtual image display area 25 to the outside scene visually recognizing area 26 (in the directions of arrows M and N) and moved to the vicinity of the boundary 27a and the boundary 27b, so that the image G4 can be enlarged. That is, the enlargement process can be performed by performing an operation of spreading both hands outward with the image G4 as the center. In the case of the reduction process, an operation opposite to the enlargement process, that is, an operation of bringing the palms of both hands closer to the virtual image display area 25 from the outside scene visual recognition area 26 side (toward the image G4 in the virtual image display area 25). Can be executed.

  In the present embodiment, various processes are executed using two indicators, but the present invention is not limited to this, and the present invention can be applied to the case where three or more indicators are used.

  In the present embodiment, the case where the size of the virtual image display area 25 is smaller than the size of the outside scene visual recognition area 26 is illustrated, but the present invention is not limited to this. When the virtual image display area 25 and the outside scene visual recognition area 26 have the same size (in other words, when the virtual image display area 25 has the same size as the visual field 21), or the virtual image display area 25 is larger than the external scene visual recognition area 26 The present invention can also be applied to cases (in other words, the virtual image display area 25 is larger than the field of view 21). In the latter case, the “predetermined display area” in the claims is the edge of the outside scene visual recognition area 26 (the edge of the visual field 21), and the “display peripheral area” is an area outside the external scene visual recognition area 26, that is, outside the visual field 21. It becomes the area of.

[Second Embodiment]
Next, a second embodiment of the present invention will be described with reference to FIGS. 15 and 16. In the first embodiment, the image display unit 40 that allows the user to visually recognize a virtual image, the control unit 30 that controls the image display unit 40, and the like are mounted in one housing (inside the device), and the head-mounted display device. In this embodiment, the head-mounted display device HM is configured by realizing and connecting these as separate devices. Hereinafter, differences from the first embodiment will be described. In the present embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and detailed description thereof is omitted. Moreover, the modification applied about the component similar to 1st Embodiment is applied similarly about this embodiment.

  FIG. 15 is an explanatory diagram showing an external configuration of a head-mounted display device HM according to the second embodiment. The head-mounted display device HM includes an image display device 300 that allows the user to visually recognize a virtual image while being mounted on the user's head, and a controller 200 (control device) that controls the image display device 300. ing. The display device in the claims has the controller 200 and the image display device 300 as main components.

  The image display device 300 displays an image in addition to the right holding unit 11, the right display driving unit 12, the left holding unit 13, the left display driving unit 14, the right optical image display unit 15, the left optical image display unit 16, and the sensor 17. A connection part 310 for connecting the apparatus 300 to the controller 200 is provided.

  The connection unit 310 includes a main body cord 311 connected to the controller 200, a right cord 312 in which the main body cord 311 branches in two, a left cord 313, and a connecting member 314 provided at a branch point. . The right cord 312 is inserted into the casing of the right holding unit 11 from the distal end AP in the extending direction of the right holding unit 11 and is connected to the right display driving unit 12. Similarly, the left cord 313 is inserted into the housing of the left holding unit 13 from the distal end AP in the extending direction of the left holding unit 13 and is connected to the left display driving unit 14.

  The image display device 300 and the controller 200 transmit various signals via the connection unit 310. Each end of the main body cord 311 opposite to the connecting member 314 and the controller 200 are each provided with a connector (not shown) that fits with each other. The connector of the main body cord 311 and the connector of the controller 200 are connected to each other. The controller 200 and the image display apparatus 300 are connected to or disconnected from each other by the fitting / releasing of. For the main body cord 311, the right cord 312, and the left cord 313, for example, a metal cable or an optical fiber can be used.

  The controller 200 is an apparatus that is connected to the image display apparatus 300 via the connection unit 310 and controls image display by the image display apparatus 300. The controller 200 includes a lighting unit 201, a touch pad 202, an operation key group 203, and a power switch 204.

  The lighting unit 201 notifies the operation state of the head-mounted display device HM according to the light emission state. The operating state of the head-mounted display device HM is, for example, an ON / OFF state of a power source. For example, an LED (Light Emitting Diode) can be used as the lighting unit 201.

  The touch pad 202 detects a user's finger operation on the operation surface and outputs a signal corresponding to the detected content. For example, a pointing (position designation) operation, a click operation, a drag operation, a flick operation, and the like by the user can be accepted.

  The touch pad 202 of this example also functions as an indicator of the first embodiment. That is, by operating the touch pad 202 with two fingers, two pointers corresponding to each finger are displayed in the virtual image display area 25, and by moving the two pointers based on the operation on the operation surface, Various commands such as display of the menu screen, enlargement / reduction processing of images, etc., rotation processing, etc. can be executed.

  The operation key group 203 includes various operation keys such as a direction key, a determination key, and a back key. Each operation key, when pressed by the user, outputs a signal indicating a process associated with the pressed operation key in advance. The power switch 204 switches the power ON / OFF of the head-mounted display device HM by detecting a slide operation of the switch.

  FIG. 16 is a block diagram functionally showing the configuration of the head-mounted display device HM according to the second embodiment. As illustrated in FIG. 16, the controller 200 includes an operation unit 211 in addition to the control unit 30, the power supply unit 71, and the interface 72. The operation unit 211 includes a touch pad 202, operation key group 203 (direction key, determination key, back key) and power switch 204.

  In addition, the CPU 50 of the control unit 30 reads out and executes a computer program from the storage unit 60 to thereby execute an operating system (OS) 51, an image processing unit 52, a display control unit 53, an image analysis unit 54, and an instruction execution unit 55. In addition to the voice processing unit 56, it also functions as the operation control unit 212.

  The operation control unit 212 receives a signal output from the operation unit 211 in response to a user operation on the operation unit 211, and causes the image processing unit 52, the display control unit 53, and the like to perform an operation according to the signal. For example, a signal indicating that the user has touched the touch pad 202 is received (detected) from the operation unit 211, and the pointer P is displayed on the image processing unit 52, the display control unit 53, or the like. That is, the operation control unit 212 of the present embodiment corresponds to a pointer detection unit in the claims.

  The image display device 300 includes the right display driving unit 12, the left display driving unit 14, the right light guide plate 86 as the right optical image display unit 15, and the left light guide plate 96 as the left optical image display unit 16. And a sensor 17, a right earphone 18, and a left earphone 19. Since the configuration of each part is the same as that of the first embodiment, the description thereof is omitted.

  Thus, the present invention can also be applied to the case where the head-mounted display device HM includes the image display device 300 that allows the user to visually recognize a virtual image and the controller 200 that controls the image display device 300. .

  In this embodiment, the controller 200 and the image display device 300 are illustrated as being connected by wire through the connection unit 310. However, the controller 200 and the image display device 300 may be connected to each other via a wireless LAN, infrared communication, or Bluetooth (registration). (Trademark) or the like.

  Further, an operation area (hereinafter referred to as “peripheral operation area”) corresponding to the outside scene visual recognition area 26 may be provided on the outer periphery (periphery) of the operation surface of the touch pad 202. As a result, for example, by moving (flicking) a finger from the operation surface of the touch pad 202 to the peripheral operation area, an operation of moving the pointer P from the virtual image display area 25 to the outside scene visual recognition area 26 across the boundary 27 is performed. Thus, by moving a finger from the peripheral operation area to the operation surface, an operation of moving the pointer P from the outside scene viewing area 26 to the virtual image display area 25 beyond the boundary 27 is possible. In other words, since the boundary between the operation surface and the peripheral operation area can be regarded as the boundary 27, an intuitive operation is possible even when the touch pad 202 is used.

  Further, the function of the controller 200 of the present embodiment may be realized by a personal computer (PC), a personal digital assistant (PAD, mobile phone, watch-type mobile terminal) or the like.

  In the above-described examples (first embodiment and second embodiment), the binocular head-mounted display device HM (optical transmission type head-mounted display device) is exemplified as the display device. The present invention is also applicable to other types of head mounted display devices such as a video transmission type, a non-transmission type, and a monocular type. In the above embodiment, the eyeglass-type head-mounted display device HM that covers the front of the eye is illustrated, but the present invention is not limited to this, and the type that does not completely cover the front of the eye (the front of the eye). The present invention can also be applied to a head-mounted display device of a type that partially covers). Furthermore, the present invention is not limited to a head-mounted display device, and can be applied to other information processing devices such as a head-up display, a personal computer, a projector, a television, and a portable information terminal.

  In the above embodiment, the image light generation unit includes a backlight control unit (right backlight control unit 81 and left backlight control unit 91), a backlight (right backlight 82 and left backlight 92), an LCD Although the control unit (the right LCD control unit 83 and the left LCD control unit 93) and the LCD (the right LCD 84 and the left LCD 94) are configured, this mode is merely an example. The image light generation unit may include a configuration unit for realizing another method together with or in place of these configuration units.

  For example, the image light generation unit may include an organic EL (Organic Electro-Luminescence) display and an organic EL control unit. Further, for example, the image light generation unit may use LCOS (Liquid crystal on silicon, LCoS is a registered trademark), a digital micromirror device, or the like instead of the LCD. Further, for example, the present invention can be applied to a laser retinal projection type head-mounted display device. In the case of the laser retinal projection type, the “image light emitting area in the image light generation unit” can be defined as an image area recognized by the user's eyes. That is, this image area corresponds to the virtual image display area 25 in the present embodiment. Further, the optical system of the head-mounted display device may be configured to allow external light to pass through the optical member and enter the user's eyes together with the image light, and may cover a part of the user's eyes. The optical member may be scanning type image light such as a laser, and the optical member is not limited to an object that guides the image light inside the member, but only has a function of refracting or reflecting the image light in the direction of the user's eyes. You may do it.

  In this embodiment, the projection optical system (the right projection optical system 85 and the left projection optical system 95) that projects the image light emitted from the image light generation unit is applied as the optical system. It is an example. For example, a scanning optical system composed of a MEMS mirror may be employed as the optical system, and a signal light modulation unit (signal light forming unit) that emits signal light to the scanning optical system may be provided. In this case, the signal light formed and emitted by the signal light modulation unit is incident on a scanning optical system (MEMS mirror) that is a scanning unit. The scanning optical system emits signal light as scanning light toward a light guide having a half mirror layer, and the scanning light is scanned on the surface of the half mirror layer to form a virtual image by image light. The user can recognize the image by grasping with the eyes. In this case, the area where the user's eyes recognize the image corresponds to the virtual image display area 25 in the present embodiment. That is, even when the scanning optical system is used, the user recognizes an image such as a pointer by the above-described method, and the execution of the command assigned near the boundary 27 of the virtual image display area 25 is controlled by the movement of the pointer. can do.

  Moreover, it is possible to provide each function (each process) of the head-mounted display device HM shown in the above embodiment as a program. Further, the program can be provided by being stored in various recording media (CD-ROM, flash memory, etc.). That is, a program for causing a computer to function as each component of the head-mounted display device HM and a recording medium on which the program is recorded are also included in the scope of the right of the present invention.

  Further, regardless of the above-described embodiments, the device configuration and processing steps of the head-mounted display device HM can be appropriately changed without departing from the gist of the present invention.

  DESCRIPTION OF SYMBOLS 15 ... Right optical image display part 16 ... Left optical image display part 17 ... Sensor 25 ... Virtual image display area 26 ... Outside view visual recognition area 27, 27a-27d ... Boundary 30 ... Control part 40 ... Image display part 52 ... Image processing part 53 ... Display control unit 54 ... Image analysis unit 55 ... Command execution unit 81 ... Right backlight control unit 82 ... Right backlight 83 ... Right LCD control unit 84 ... Right LCD 86 ... Right light guide plate 91 ... Left backlight control unit 92 ... Left Backlight 93 ... Left LCD controller 94 ... Left LCD 96 ... Left light guide plate 200 ... Controller 300 ... Image display device P, P1, P2 ... Pointer IC ... Icon HM ... Head-mounted display device

Claims (16)

An indicator detection unit for detecting a plurality of indicators operated by the user;
A pointer display unit that detects each indicator by the indicator detection unit and displays a pointer that operates in conjunction with each of the indicators in a predetermined display area;
When the plurality of pointers move to the vicinity of the boundary between the predetermined display area and a display peripheral area other than the predetermined display area in accordance with the operation of each indicator, the plurality of pointers An instruction execution unit that executes a predetermined instruction according to the movement of the display device. The instruction execution unit is
The display device according to claim 1, wherein different instructions are executed according to a movement order of the plurality of pointers to the vicinity of the boundary. The instruction execution unit is
3. The display device according to claim 1, wherein a frame image is displayed along an inner edge of the display area when at least one of the plurality of pointers moves in the vicinity of the boundary. . The instruction execution unit is
When the indicator detection unit detects that the plurality of indicators are operated in a direction to move from the predetermined display area to the display peripheral area beyond the boundary, a predetermined instruction is executed. The display device according to claim 1, wherein the display device is a display device. The instruction execution unit is
When the indicator detection unit detects that the plurality of indicators are operated from the display peripheral area to the predetermined display area across the boundary, a predetermined command is executed. The display device according to claim 1, wherein the display device is characterized. The instruction execution unit is
The predetermined instruction is executed when the pointer detection unit detects an operation of the plurality of pointers in which the plurality of pointers move along the boundary. Item 1. A display device according to item 1. The instruction execution unit is
The display device according to claim 6, wherein different instructions are executed according to a moving direction of the plurality of pointers when the plurality of pointers move along the boundary. The instruction execution unit is
When at least one indicator among the plurality of indicators is rotated in a state where the plurality of pointers are in the vicinity of the boundary, an image displayed in the predetermined display area is displayed in a rotation direction of the indicator. The display device according to claim 1, wherein the display device is rotated. The instruction execution unit is
The display device according to claim 1, wherein a predetermined command is executed in accordance with an operation of each indicator in the vicinity of the boundary. The instruction execution unit is
When the indicator detection unit detects an operation in which the plurality of indicators sandwich the icon displayed in the predetermined display area across the boundary from the display peripheral area, a predetermined command assigned to the icon The display device according to claim 1, wherein the display device is executed. The instruction execution unit is
The display device according to any one of claims 1 to 10, wherein a predetermined command corresponding to the displayed content is executed. The display device according to any one of claims 1 to 11,
The display device is a transmissive display device that allows a user to visually recognize an image as a virtual image,
The indicator detection unit includes:
A display device, wherein a user detects the plurality of indicators that are operated in front of the outside of the display device that is opposite to a user side of the transmissive display device. The display device according to any one of claims 1 to 11,
The display device is a transmissive display device that allows a user to visually recognize an image as a virtual image,
The image display unit of the transmissive display device includes:
An image light generation unit that generates and emits image light representing an image using image data to be displayed; and
An optical member for guiding the emitted image light to the eyes of the user,
A display device characterized in that light of an outside scene passes through the optical member and enters the user's eyes together with the image light. A transmissive head-mounted display device that allows a user to visually recognize an image as a virtual image,
An indicator detection unit for detecting a plurality of indicators operated by the user;
A pointer display unit that detects each indicator by the indicator detection unit and displays a pointer that operates in conjunction with each of the indicators in a predetermined display area;
When the plurality of pointers move to the vicinity of the boundary between the predetermined display area and a display peripheral area other than the predetermined display area in accordance with the operation of each indicator, the plurality of pointers And a command execution unit that executes a predetermined command according to the movement of the head. The indicator detection unit includes:
The head-mounted display device according to claim 14, wherein the plurality of indicators are detected by imaging the front of the user in a state where the head-mounted display device is mounted. Display device
Detecting a plurality of indicators operated by the user;
Detecting each indicator and displaying a pointer that operates in conjunction with each of the indicators in a predetermined display area;
When the plurality of pointers move to the vicinity of the boundary between the predetermined display area and a display peripheral area other than the predetermined display area in accordance with the operation of each indicator, the plurality of pointers And a step of executing a predetermined command in accordance with the movement of the display device.

Images