JP6432583B2 - Display device, head-mounted display device, and display device control method - Google Patents

Description

  The present invention relates to a display device that can be operated by an indicator operated by a user, a head-mounted display device, and a display device control method.

  2. Description of the Related Art Conventionally, as a means for operating a head-mounted display device (Head Mounted Display (HMD)), which is a display device mounted on the head, a predetermined motion of a human finger is detected and the motion is handled. What performs processing is known. For example, in Patent Document 1, an input operation corresponding to the movement of a finger is determined by imaging an LED (light emitting body) mounted on the user's finger with a video camera, and a predetermined process is performed based on the determined input operation. Is disclosed.

JP 2000-29619 A

  However, in the case of the above-described prior art, control is complicated because it is necessary to recognize a complicated movement of the user's finger with a video camera. In addition, there is a problem that the types of control are limited or misrecognized easily despite the complicated mechanism. Furthermore, there is a problem that the user has to remember what kind of processing the movement of the finger is to perform, and surely perform the movement, which is not always convenient.

  In view of the above-described problems, the present invention provides a display device, a head-mounted display device, and a display device control method capable of providing a user interface capable of executing various processes with simple and intuitive operations. The purpose is to do.

The display device of the present invention includes a pointer display unit that displays a pointer that operates in conjunction with an indicator operated by a user in a predetermined display area, and the pointer is an area other than the predetermined display area from the predetermined display area. An instruction execution unit that executes an image enlargement / reduction process when the indicator is operated so as to move to the display peripheral region, and the pointer display unit displays the pointer until the enlargement / reduction process is performed. After the display is performed in the shape and the enlargement / reduction processing is performed, the display is performed in a second shape different from the first shape .
In the above display device, the pointer display unit performs the image enlargement process as the enlargement / reduction process. As a result, the image cannot be enlarged any more, or as a result of the image reduction process as the enlargement / reduction process. The pointer is changed to a third shape that is different from both the first shape and the second shape after the state becomes unable to be further reduced .
In the above display device, the instruction execution unit may perform an image scaling ratio according to at least one of a speed at which the pointer contacts the boundary between the predetermined display area and the display peripheral area, the number of times of contact, and a contact angle with the boundary. It is characterized by controlling.
In the above display device, the instruction execution unit executes an image enlargement / reduction process when the pointer moves to the vicinity of a portion to which an instruction is assigned in the boundary between the predetermined display area and the display peripheral area, The pointer display section includes a pointer when the pointer moves to the vicinity of the portion to which the instruction is assigned in the boundary, and a pointer when the pointer moves to the vicinity of the portion of the boundary to which the instruction is not assigned. Are displayed in different shapes.
In the above display device, the display device is a transmissive display device that allows a user to visually recognize an image as a virtual image, and the user is on the outer front side of the display device that is opposite to the user side of the transmissive display device. It is characterized in that an indicator operated by the is detected.
In the above display device, the display device is a transmissive display device that allows a user to visually recognize an image as a virtual image, and the transmissive display device generates image light representing an image using image data to be displayed. And an optical member that guides the emitted image light to the user's eye, and allows the light of the outside scene to pass through the optical member and enter the user's eye together with the image light. It is characterized by.
A head-mounted display device according to the present invention is a transmissive head-mounted display device that allows a user to visually recognize an image as a virtual image, and a pointer that operates in conjunction with an indicator operated by the user is a predetermined pointer. A pointer display unit to be displayed in the display area, and a command for executing an image scaling process when the pointer is operated so that the pointer moves from a predetermined display area to a display peripheral area other than the predetermined display area An execution unit, and the pointer display unit displays the pointer in a first shape until the enlargement / reduction process is performed, and after the enlargement / reduction process is performed, a second different from the first shape is displayed. It is characterized by being displayed in shape .
In the head-mounted display device described above, the indicator is detected by capturing an image of the front of the user in a state in which the head-mounted display device is mounted.
The display device control method of the present invention includes a pointer display step in which a display device displays a pointer that operates in conjunction with an indicator operated by a user in a predetermined display area, and the pointer is moved from a predetermined display area to a predetermined area. When the indicator is operated to move to a display peripheral area that is an area other than the display area, an instruction execution step for executing image enlargement / reduction processing is executed. In the pointer display step, the pointer is Until it is performed, the image is displayed in the first shape, and after the enlargement / reduction processing is performed, the image is displayed in a second shape different from the first shape .
The following configuration may be used.
A display device according to the present invention includes a pointer detection unit that detects a pointer operated by a user, and a pointer that operates in conjunction with the pointer using a detection of the pointer by the pointer detection unit as a trigger. When the pointer moves to the vicinity of the boundary between the predetermined display area and the display peripheral area other than the predetermined display area in accordance with the operation of the pointer display unit displayed on the indicator and the indicator, the predetermined instruction And an instruction execution unit for executing.

  The head-mounted display device of the present invention is a transmissive head-mounted display device that allows a user to visually recognize an image as a virtual image, and includes an indicator detection unit that detects an indicator operated by the user, A pointer display unit that displays a pointer that operates in conjunction with the indicator in a predetermined display area with the detection of the indicator by the body detection unit as a trigger, and the pointer is moved to a predetermined display area in accordance with the operation of the indicator And an instruction execution unit that executes a predetermined instruction when moved to the vicinity of the boundary with the display peripheral area that is an area other than the predetermined display area.

  According to the display device control method of the present invention, the display device detects a pointer operated by the user, and a pointer that operates in conjunction with the pointer is triggered by the detection of the pointer in a predetermined display area. When the pointer moves to the vicinity of the boundary between the predetermined display area and the display peripheral area other than the predetermined display area in accordance with the display step and the operation of the indicator, the predetermined instruction is executed. And performing the steps.

  According to these configurations, it is possible to execute a predetermined command (processing) assigned in advance near the boundary simply by operating the indicator so that the user moves the pointer near the boundary of the display area. That is, since the user can execute a predetermined command by a simple (simple) and intuitive operation without performing a complicated operation, it is easy to use. Further, since it can be performed with a simple operation (no complicated operation is required), it is possible to provide a user interface that operates stably and has few misrecognitions.

  In the display device of the present invention, it is preferable that the predetermined command is assigned corresponding to a position near the boundary.

  According to this configuration, since various instructions can be assigned according to the position where the pointer is moved, it is possible to provide a user interface that can respond to various needs of the user. For example, when the display area is a rectangle, the same command may be assigned in the vicinity of the upper, lower, left, and right boundaries, or different commands may be assigned. Alternatively, even in the vicinity of the right boundary, a different command may be assigned for each position such as an upper part and a lower part of the boundary.

  In the display device of the present invention, when the instruction execution unit detects that the pointer is operated in a direction in which the pointer moves from the predetermined display area to the display peripheral area beyond the boundary by the indicator detection unit, It is preferable to execute these instructions.

  In the display device of the present invention, it is preferable that the command execution unit executes different commands depending on the moving direction of the indicator when the pointer moves from a predetermined display area in a direction beyond the boundary.

  According to these configurations, when the user performs an operation for moving the pointer in a direction beyond the boundary, a predetermined command is executed. In addition, by executing different commands depending on the movement direction of the pointer with respect to the boundary (for example, whether the operation is to move obliquely upward or diagonally downward with respect to the boundary), A high (high flexibility) user interface can be provided.

  In the display device of the present invention, it is preferable that the command execution unit executes a predetermined command when the pointer detection unit detects an operation of the pointer whose pointer moves along the boundary.

  In the display device of the present invention, it is preferable that the instruction execution unit executes different instructions depending on a moving direction when the pointer moves along the boundary.

  According to these configurations, when the user performs an operation of moving the pointer along the boundary, a predetermined command is executed. Further, by executing different instructions depending on the direction of movement along the boundary (for example, whether the movement is upward or downward along the boundary), the scalability is high ( A user interface with a high degree of freedom can be provided.

  In the display device of the present invention, it is preferable that the command execution unit executes a predetermined command corresponding to the displayed content.

  According to this configuration, for each content to be displayed, a command suitable for the content can be executed. Note that the content means information content including an image (still image, moving image), sound, or the like, an application, or the like.

  In the display device of the present invention, when the pointer moves to the vicinity of the boundary between the predetermined display area and the display peripheral area, it is preferable to change the shape of the pointer.

  According to this configuration, it is possible to notify the user that the pointer has moved to the vicinity of the boundary (that is, the position where the predetermined command is executed) depending on the shape of the pointer.

  In the display device of the present invention, the predetermined command is preferably a command for displaying an icon.

  According to this configuration, an icon is displayed when the pointer moves to the vicinity of the boundary. For example, when the display device is a transmissive device that allows a user to view a virtual image and an outside scene at the same time, if the icon is always displayed, the icon obstructs the user's view (the view It is possible to get in the way. For this reason, for example, an icon is not normally displayed but an icon is displayed triggered by the operation of the pointer, so that the user's field of view can be prevented from being hindered by the icon.

  In the display device described above, the display device is a transmissive display device that allows a user to visually recognize an image as a virtual image, and the indicator detection unit is configured so that the user is opposite to the user side of the transmissive display device. It is characterized in that an indicator operating on the outer front side of the display device on the side is detected.

  According to this configuration, in the transmissive display device, it is possible to detect the indicator operated outside (front) beyond the display device as viewed from the user.

  In the display device described above, the display device is a transmissive display device that allows a user to visually recognize an image as a virtual image, and the image display unit of the transmissive display device uses image data to be displayed. An image light generation unit that generates and emits image light that represents the image, and an optical member that guides the emitted image light to the user's eyes, and transmits light of the outside scene through the optical member and the image light It is made to enter into a user's eyes.

  According to this configuration, the present invention can also be applied to a transmissive display device in which a user can view a virtual image and an outside scene at the same time.

  In the head-mounted display device of the present invention, it is preferable that the indicator detection unit detects the indicator by imaging the front of the user in a state where the head-mounted display device is mounted.

  According to this configuration, the indicator operated by the user is detected based on the imaging result in front of the user when the head-mounted display device is mounted. In other words, the user can wear the head-mounted display device and image and detect the indicator operated in front of the eyes (in front of the device).

It is explanatory drawing which shows the external appearance structure of the head mounted display apparatus which concerns on 1st Embodiment. It is a figure explaining an example of a virtual image and an outside scene which a user can recognize visually. It is a block diagram explaining the functional structure of the head mounted display apparatus which concerns on 1st Embodiment. It is a figure explaining the case where the process which displays a menu screen by operation of a pointer is performed. It is a figure explaining an example of the menu screen displayed by operation of a pointer. It is a figure explaining the case where an expansion process is performed. It is a figure explaining the case where a reduction process is performed. It is a figure explaining the case where a page feed process is performed. It is a figure explaining the case where a page return process is performed. It is a figure explaining the case where a rotation process is performed. It is a figure explaining the case where an expansion process and a rotation process are performed simultaneously. It is a figure explaining the case where a reduction process and a rotation process are performed simultaneously. It is explanatory drawing which shows the external appearance structure of the head mounted display apparatus which concerns on 2nd Embodiment. It is a block diagram explaining the functional structure of the head mounted display apparatus which concerns on 2nd Embodiment.

[First Embodiment]
Hereinafter, a display device, a head-mounted display device, and a display device control method according to an embodiment of the present invention will be described with reference to the accompanying drawings. FIG. 1 is an explanatory diagram showing an external configuration of a head-mounted display device HM as a display device according to an embodiment of the present invention. The head-mounted display device HM is a display device that a user wears on the head, and is also referred to as a head mounted display (HMD). The head-mounted display device HM of this embodiment is an optically transmissive head-mounted display device HM (a transmissive head-mounted display device, a see-through type) that allows a user to visually recognize a virtual image and at the same time directly view an outside scene. -A head-mounted display) having a spectacle shape.

  The head-mounted display device HM includes a right holding unit 11, a right display driving unit 12, a left holding unit 13, a left display driving unit 14, a right optical image display unit 15, and a left optical image display unit 16. , Sensor 17.

  The right optical image display unit 15 and the left optical image display unit 16 are disposed at positions corresponding to the right and left eyes of the user when the head-mounted display device HM is mounted. One end of the right optical image display unit 15 and one end of the left optical image display unit 16 are connected at positions corresponding to the user's eyebrows when the head-mounted display device HM is mounted. The right holding unit 11 extends from the end ER that is the other end of the right optical image display unit 15. Similarly, the left holding unit 13 extends from the end EL which is the other end of the left optical image display unit 16.

  The right holding unit 11 is substantially perpendicular to the right optical image display unit 15 from the end ER of the right optical image display unit 15 to a position corresponding to the user's temporal region when the head-mounted display device HM is mounted. It is a member provided by stretching so as to be formed. Similarly, the left holding unit 13 is connected to the left optical image display unit 16 from the end EL of the left optical image display unit 16 to a position corresponding to the user's temporal region when the head-mounted display device HM is worn. It is a member provided by extending so as to form a substantially right angle. The right holding unit 11 and the left holding unit 13 hold the head-mounted display device HM on the user's head like a temple of glasses.

  The right display drive unit 12 is located inside the right holding unit 11, in other words, on the side facing the user's head when the head-mounted display device HM is worn, and the end of the right optical image display unit 15. It is arranged on the part ER side. The left display driving unit 14 is disposed inside the left holding unit 13 and on the end EL side of the left optical image display unit 16.

  The sensor 17 is an imaging device arranged at a position corresponding to the user's eyebrow when the head-mounted display device HM is worn. The sensor 17 captures an image of an outside scene in the viewing direction of the user (in front of the user) in a state where the head-mounted display device HM is worn, and acquires an image including an indicator operated by the user. The image signal of the image is output. That is, the indicator operated on the side opposite to the user side in the head-mounted display device HM (outside front of the head-mounted display device HM) is imaged. The imaging range by the sensor 17 of this embodiment is wider than the user's field of view 21 (see FIG. 2) when the head-mounted display device HM is worn.

  As the indicator, for example, a user's fingertip, a ring-type input device worn by the user, a pen nib of the user, a light emitter of a remote controller, or the like can be used. When the user's fingertip or pen tip is used as the indicator, for example, the finger tip or pen tip may be recognized as a feature point (specific point) and the position may be detected. When a ring type input device is used, for example, an infrared light emitter (LED or the like) may be attached to the ring type input device, and the position may be detected by the infrared light, or the ring type input device is attached. The user's fingertip may be recognized as a feature point to detect the position. Furthermore, an arbitrary object virtually held by the user can be used as the indicator. “Anything that is virtually held” means, for example, a pen that is virtually assumed to follow the hand of the user when the shape of the user's hand becomes a shape of holding a pen. Even in such a case, similarly to the above, an arbitrary place of a virtually held object (for example, a virtually assumed pen nib) is recognized as an indicator, and the pen tip The locus or the like can be recognized as a feature point, and the position can be detected.

  In this embodiment, one sensor 17 is illustrated, but a plurality of sensors 17 may be provided. In the present embodiment, an imaging device (imaging sensor) is illustrated as the sensor 17, but the present invention is not limited to this, and the type (detection method) is not limited as long as the indicator can be detected. For example, a laser sensor, an ultrasonic sensor, or the like can be used as the sensor 17, and a plurality of types of detection methods may be combined.

  The head-mounted display device HM further includes a right earphone 18 for the right ear and a left earphone 19 for the left ear. The right earphone 18 and the left earphone 19 are respectively attached to the right and left ears when the user wears the head-mounted display device HM.

  Next, an example of a virtual image and an outside scene that can be visually recognized by the user wearing the head-mounted display device HM will be described with reference to FIG. As shown in FIG. 2, a virtual image 22 generated by the head-mounted display device HM is displayed in the field of view 21 of the user wearing the head-mounted display device HM. Further, the user can see the outside scene 23 through the right optical image display unit 15 and the left optical image display unit 16 in a region other than the region where the virtual image 22 is displayed in the user's field of view 21. (Hereinafter, an area where the virtual image 22 is displayed is referred to as a “virtual image display area 25”, and an area other than this is referred to as an “outside view visually recognizing area 26”). Further, in the head-mounted display device HM of the present embodiment, the outside scene 23 can be seen through the virtual image 22 in the area (virtual image display area 25) in which the virtual image 22 is displayed in the user's field of view 21. It is like that. Although details will be described later, various commands for executing predetermined processing are allocated in the vicinity of the boundary 27 (the edge of the virtual image display area 25) between the virtual image display area 25 and the outside scene visual recognition area 26. It can be executed by the operation of the indicator by the user.

  Next, the functional configuration of the head-mounted display device HM will be described with reference to FIG. As shown in FIG. 3, the head-mounted display device HM includes an image display unit 40 that allows a user to visually recognize a virtual image, a sensor 17, a right earphone 18, a left earphone 19, a power supply unit 71, and an interface 72. And a control unit 30 that controls these units.

  The power supply unit 71 supplies power to each unit of the head-mounted display device HM. The interface 72 is for connecting various external devices OA that are sources of contents such as images (still images, moving images) and audio. Examples of the external device OA include a personal computer (PC), a mobile terminal (such as a mobile phone and a smartphone), and a game terminal. As the interface 72, for example, a USB interface, a micro USB interface, a memory card interface, a wireless LAN interface, or the like can be employed.

  The control unit 30 includes a CPU (Central Processing Unit) 50 and a storage unit 60. The storage unit 60 includes a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and stores various computer programs. The CPU 50 reads out and executes a computer program from the storage unit 60, thereby operating as an operating system (OS) 51, an image processing unit 52, a display control unit 53, an image analysis unit 54, a command execution unit 55, and a voice processing unit 56. Function.

  The image processing unit 52 generates a signal to be supplied to the image display unit 40 based on content input through the interface 72 and transmits the signal to the image display unit 40. The signal supplied to the image display unit 40 differs between the analog format and the digital format.

  For example, in the case of an analog format, the image processing unit 52 generates and transmits a clock signal PCLK, a vertical synchronization signal VSync, a horizontal synchronization signal HSync, and image data Data. Specifically, the image processing unit 52 acquires an image signal (for example, an analog signal composed of 30 frame images per second) included in the content, and from the acquired image signal, a vertical synchronization signal VSync, A synchronization signal such as the horizontal synchronization signal HSync is separated. Then, the image processing unit 52 generates a clock signal PCLK using a PLL circuit (not shown) or the like according to the separated vertical synchronization signal VSync, horizontal synchronization signal HSync, and the like. Further, the image processing unit 52 converts the analog image signal from which the synchronization signal has been separated into a digital image signal using an A / D conversion circuit (not shown) or the like to generate image data Data.

  On the other hand, in the case of the digital format, the image processing unit 52 generates and transmits the clock signal PCLK and the image data Data. Specifically, when the content is in a digital format, the clock signal PCLK is output in synchronization with the image signal. For this reason, A / D conversion of the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the analog image signal becomes unnecessary.

  Note that the image processing unit 52 may execute image processing such as resolution conversion processing, color tone correction processing such as luminance / saturation adjustment, and keystone correction processing on the image data Data as necessary.

  The display control unit 53 controls the image display state in the image display unit 40 by generating control signals for controlling the right display drive unit 12 and the left display drive unit 14 and transmitting them to the image display unit 40. Specifically, the display control unit 53 uses the control signal to turn on / off the drive of the right LCD 84 by the right LCD control unit 83, drive the ON / OFF of the right backlight 82 by the right backlight control unit 81, and the left LCD. The right display drive unit 12 and the left display drive unit 14 are individually controlled by controlling ON / OFF of the left LCD 94 by the control unit 93 and ON / OFF of the left backlight 92 by the left backlight control unit 91, respectively. The generation and emission of image light by each of the above are controlled. For example, the display control unit 53 causes both the right display driving unit 12 and the left display driving unit 14 to generate image light, generates only one image light, or does not both generate image light.

  The image analysis unit 54 acquires an image signal of an image captured by the sensor 17 and analyzes the acquired image signal. The image analysis unit 54 detects the indicator by image analysis and outputs position information thereof. The image processing unit 52 and the display control unit 53 cause the image display unit 40 to display a pointer P (see FIG. 4A) corresponding to the indicator based on the output positional information of the indicator. In addition, the indicator detection part in a claim has the sensor 17 and the image analysis part 54 as main components. Further, the pointer display unit in the claims has the image processing unit 52 and the display control unit 53 as main components.

  The command execution unit 55 executes various commands according to the movement position and movement direction of the pointer P accompanying the operation of the indicator. Specifically, when the pointer P moves to the vicinity (both see FIG. 2) of the boundary 27 between the virtual image display area 25 and the outside scene visual recognition area 26, or when the pointer P crosses the boundary 27 from the virtual image display area 25 to the outside scene. Various instructions (processing) assigned in advance are executed when the indicator is operated in the direction of moving to the visual recognition area 26 or when the pointer P is moved along the boundary 27 (details will be described later). . That is, in the present embodiment, “predetermined display area” in the claims corresponds to the virtual image display area 25, and “display peripheral area” corresponds to the outside scene visual recognition area 26. The boundary 27 between the virtual image display area 25 and the outside scene visually recognizing area 26 indicates a limit at which the pointer P can be displayed (the pointer P can be moved).

  The audio processing unit 56 acquires an audio signal included in the content, amplifies the acquired audio signal, and transmits the amplified audio signal to the right earphone 18 and the left earphone 19.

  The image display unit 40 includes a right display drive unit 12, a left display drive unit 14, a right light guide plate 86 as the right optical image display unit 15, and a left light guide plate 96 as the left optical image display unit 16. doing. In the following description, the right optical image display unit 15 and the right light guide plate 86, and the left optical image display unit 16 and the left light guide plate 96 are also collectively referred to as “optical members”.

  A right display drive unit 12, a right backlight (BL) control unit 81 and a right backlight (BL) 82 functioning as a light source, a right LCD control unit 83 and a right LCD 84 functioning as display elements, and a right projection optical system 85 Contains. In the following description, the right backlight control unit 81, the right backlight 82, the right LCD control unit 83, and the right LCD 84 are also collectively referred to as “image light generation unit”.

  The right backlight control unit 81 drives and controls the right backlight 82 based on the input control signal. The right backlight 82 is a light emitter such as an LED or electroluminescence (EL). The right LCD control unit 83 drives and controls the right LCD 84 based on the input clock signal PCLK, vertical synchronization signal VSync, horizontal synchronization signal HSync, and image data Data for the right eye. The right LCD 84 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix.

  The right projection optical system 85 projects image light emitted from the right LCD 84, and is configured using, for example, a collimator lens. The right light guide plate 86 as the right optical image display unit 15 is formed of a light-transmitting resin material or the like, and reflects the image light output from the right projection optical system 85 along a predetermined optical path while being reflected by the user. Lead to the right eye. The right projection optical system 85 and the right light guide plate 86 are collectively referred to as “light guide unit”. The light guide unit can use any method as long as it forms a virtual image in front of the user's eyes using image light. For example, a diffraction grating (hologram element as an example) may be used, or transflective reflection may be used. A film may be used.

  As with the right display drive unit 12, the left display drive unit 14 includes a left backlight (BL) control unit 91, a left backlight (BL) 92, a left LCD control unit 93, a left LCD 94, and left projection optics. (Like the right display drive unit 12, the left backlight control unit 91, the left backlight 92, the left LCD control unit 93, and the left LCD 94 are collectively referred to as an “image light generation unit”. Call). Since the configuration and function of each element included in the left display driving unit 14 are the same as those included in the right display driving unit 12, the description thereof is omitted here. The left light guide plate 96 as the left optical image display unit 16 is formed of a light transmissive resin material or the like, and is used while reflecting the image light output from the left projection optical system 95 along a predetermined optical path. To the left eye.

  In this way, the image light guided to both eyes of the user of the head-mounted display device HM forms an image on the retina, so that the user can enter the virtual image display area 25 as shown in FIG. The virtual image 22 is visually recognized.

  Next, referring to FIGS. 4 to 12, various commands are executed by the movement of the pointer P displayed in the virtual image display area 25 by the head-mounted display device HM in accordance with the operation of the indicator by the user. An example of the procedure will be described. 4 and 5 are diagrams illustrating a case where a process for displaying a menu screen is performed by an operation of the indicator (movement of the pointer P). 6 and 7 are diagrams for explaining a case where enlargement processing or reduction processing of an image or the like is executed. 8 and 9 are diagrams for explaining page movement processing (page feed processing and page return processing) for document images and the like. FIG. 10 is a diagram for explaining a case where image rotation processing is executed. FIGS. 11 and 12 are diagrams for explaining a case where enlargement / reduction processing and rotation processing of an image or the like are executed simultaneously.

  First, with reference to FIG. 4 and FIG. 5, an example of a procedure (a user interface for displaying a menu screen) when executing a menu screen display process will be described. As shown in FIG. 4A, when the indicator operated by the user is detected in the virtual image display region 25 (more precisely, the indicator is in the imaging region of the sensor 17 overlapping the virtual image display region 25). The head-mounted display device HM displays a pointer P (circular) in the virtual image display area 25 based on the detection position of the indicator. Further, the head-mounted display device HM can execute a command assigned in the vicinity of the boundary 27 between the virtual image display area 25 and the outside scene visual recognition area 26 (hereinafter referred to as “input standby mode”). Execute the process to transit to.

  When the indicator is detected outside the virtual image display area 25, the head-mounted display device HM displays, for example, an arrow-shaped pointer indicating the detection direction of the indicator, or displays the pointer in a dotted line. Then, the user is notified that the indicator is outside the virtual image display area 25. In this case, the head-mounted display device HM does not transition to the “input standby mode”.

  Next, as shown in FIG. 4B, in the input standby mode, the pointer P is near the boundary 27 by the operation of the indicator by the user (in FIG. 4B, as an example, the virtual image display area 25 The head-mounted display device HM moves to the inside of the virtual image display area 25 in the vicinity of the boundary 27 (the boundary 27 between the virtual image display area 25 and the outside scene visual recognition area 26). Executes a process for transitioning to a mode for accepting an instruction assigned to the vicinity) (hereinafter referred to as “instruction acceptance mode”) (executes a transition instruction to the instruction acceptance mode), and makes the shape of the pointer P semicircular The deformed pointer P1 is displayed, and a frame 101 (shaded portion in the drawing) is displayed along the boundary 27 (along the virtual image display area 25 side of the boundary 27 (inside the virtual image display area 25)). By deforming the shape of the pointer P and displaying the frame 101, the user can recognize that the command acceptance mode has been entered. When the semicircular pointer P1 is moved to a position away from the vicinity of the boundary 27 of the virtual image display area 25 (when moved toward the center of the virtual image display area 25), the shape thereof is the original circle. Return to the pointer P.

  Here, since the “menu screen display command” for displaying the menu screen is assigned in the vicinity of the boundary 27a, the head-mounted display device HM simultaneously performs “transition to the command reception mode” as described above. The “menu screen display command” is executed, and a menu screen D01 including various icons IC is displayed in the virtual image display area 25 as shown in FIG. In FIG. 4C, only the menu screen D01 including the icon IC is displayed. For example, other virtual images (for example, contents such as images and videos) are displayed behind the menu screen D01. (The menu screen D01 may be displayed so as to be superimposed on another virtual image).

  In the above description, the process of transitioning to the “command reception mode” is executed by moving the pointer P to the vicinity of the right boundary 27a of the virtual image display area 25. However, this is merely an example, and the present invention is not limited to this. Absent. For example, when the pointer P moves to the vicinity of the left boundary 27b, the vicinity of the upper boundary 27c, or the vicinity of the lower boundary 27d, it is possible to execute the process of transitioning to the “command acceptance mode”. is there.

  In the above description, it is assumed that the “menu screen display command” is assigned to the vicinity of the right boundary 27a of the virtual image display area 25. However, this is merely an example, for example, the vicinity of the left boundary 27b, the upper boundary It is also possible to assign an instruction to the vicinity of 27c, the vicinity of the lower boundary 27d, or the like (that is, the position to which the instruction is assigned does not matter if it is in the vicinity of the boundary 27). When no instruction is assigned in the vicinity of the boundary 27a (27), the shape of the pointer P is displayed with a prohibition mark to notify the user of that fact.

  In FIG. 4B, the frame 101 is shaded for the sake of convenience, but actually, the frame 101 is displayed in a color that can be identified by the user (for example, white, gray, red, etc.). It ’s fine. Further, the frame 101 may be displayed as a dotted line instead of a solid line, or may be without a frame line. In FIG. 4B, when the mode is changed to the “command acceptance mode”, the circular pointer P is displayed as a semicircular pointer P1, but this is not restrictive. For example, a pointer with a larger size of the pointer P may be displayed, or a double round pointer may be displayed. The same applies to the following description.

  Further, the arrangement of the icon ICs on the menu screen D01 in FIG. 4C is merely an example, and the present invention is not limited to this. 4C illustrates the menu screen D01 in which the icon IC is arranged on the left end side (and the corner) of the virtual image display area 25. For example, the icon IC is displayed on the right end side of the virtual image display area 25. An arranged menu screen may be displayed, or a menu screen in which icon ICs are arranged at both left and right ends may be displayed. Alternatively, a menu screen in which the icon IC is arranged near the center of the virtual image display area 25 may be displayed.

  Furthermore, a menu screen D02 in which a plurality of icon ICs as shown in FIG. 5 are arranged in a ring shape may be displayed as the menu screen. The menu screen D02 allows a desired icon IC to be selected by rotating the icon IC in the circumferential direction of the ring by a predetermined rotation operation. Further, the icon group ICG to be displayed can be switched by designating the mark of the indicator 102 arranged on the left side. FIG. 5A is an example of the menu screen D02 when the mark of the top indicator 102 is designated, and FIG. 5B is an example of the menu screen D02 when the mark of the second indicator 102 is designated. It is.

  The menu screen to be displayed may be switched according to the setting status of the head-mounted display device HM. For example, when the head-mounted display device HM is set to a mode intended for outdoor use, a menu screen D01 in which an icon IC is arranged at the end of the virtual image display area 25 as shown in FIG. When it is set to a mode intended for indoor use (for example, a mode for watching movies, etc.) A menu screen D02 as shown in FIG. 5 may be displayed.

  Next, an example of a procedure (user interface for executing the enlargement / reduction process) when executing the enlargement / reduction process (enlargement / reduction instruction) of the image will be described with reference to FIGS. 6 and 7. The procedure from the detection of the indicator to the transition to the command reception mode is the same as described above, and the description thereof is omitted. The same applies to the following description.

  As shown in FIG. 6A, after the user has designated the range 111 to be enlarged in the image G1 in the state of transition to the command acceptance mode, the pointer P1 is moved to the right boundary 27a by the operation of the indicator. On the other hand, when it is operated to move from the virtual image display area 25 to the outside scene visually recognizing area 26 in the diagonally upper right direction (when the indicator is operated in the direction of the arrow A. However, the pointer P1 is a virtual image. The head-mounted display device HM does not move outside the display area 25, and the head-mounted display device HM enlarges the image in the specified range 111 at a predetermined magnification (enlargement command) as shown in FIG. ) And an enlarged image 112 as a result of the execution is displayed.

  At this time, the head-mounted display device HM changes the shape of the pointer P1 to a figure (pointer P2) indicating that the enlargement process has been performed, and the enlargement process is performed at a predetermined position in the virtual image display area 25. A plus sign 113 is displayed to indicate this. In addition, when the user reaches the limit of the enlargement process by repeating the enlargement operation (when it becomes impossible to enlarge any more), the head-mounted display device HM changes the pointer to the shape of the prohibition mark. And display.

  As shown in FIG. 6C, when the indicator is operated by the user so that the pointer P1 moves upward along the right boundary 27a (indicator (pointer P1 in the direction of arrow B)). ) Is also moved), the enlargement process (enlargement command) can be executed in the same manner as described above.

  On the other hand, as shown in FIG. 7A, after the user has designated the range 111 to be reduced in the image G1 in the state of transition to the command acceptance mode, the pointer P1 is moved to the right boundary by the operation of the indicator. When operated to move to the outside view visual recognition area 26 from the virtual image display area 25 across the boundary 27a in the diagonally lower right direction with respect to 27a (when the indicator is operated in the direction of arrow C), As shown in FIG. 7B, the wearable display device HM executes a reduction process (reduction command) for reducing an image in a specified range 111 at a predetermined magnification, and a reduced image 114 as a result of the execution is displayed. indicate.

  At this time, the head-mounted display device HM changes the shape of the pointer P <b> 1 to a figure (pointer P <b> 3) indicating that the reduction process has been performed, and shows the reduction process at a predetermined position in the virtual image display area 25. A minus sign 115 is displayed. In addition, when the user reaches the limit of the reduction process by repeating the reduction operation (when the reduction cannot be performed any more), the head-mounted display device HM changes the pointer to the shape of the prohibition mark. Change and display.

  As shown in FIG. 7C, when the pointer is operated by the user so that the pointer P1 moves downward along the right boundary 27a (the pointer (pointer P1 in the direction of arrow D)). ) Is also moved), the reduction process (reduction instruction) can be executed in the same manner as described above.

  In the above description, the enlargement process and the reduction process are performed on the range 111 specified by the user. For example, the entire image G1 can be enlarged or reduced without specifying the range. In the above description, in order to notify the user that the enlargement process or the reduction process has been performed, the shape of the semicircular pointer P1 is changed to a shape corresponding to the executed process (the shape of the pointer P2 or the pointer P3). However, the pointer P1 may be changed to a shape (graphic) different from the above.

  In the above description, in order to notify the user that the enlargement process or the reduction process has been performed, the plus sign 113 indicating the enlargement process and the minus sign 115 indicating the reduction process are displayed. However, the present invention is not limited to this. Absent. For example, an indicator that can recognize the enlargement process or the reduction process may be displayed. That is, as long as it is possible for the user to recognize the processing contents, the shape and means are not limited. The same applies to the following description.

  In the above description, the enlargement / reduction of the image G1 has been described. For example, when text input is performed using a word processor or the like, the text input screen may be enlarged or reduced by the same operation as described above. Is possible.

  Next, with reference to FIGS. 8 and 9, an example of a procedure (user interface for page movement processing) in the case of executing page movement processing (page movement instruction) for performing page feed and page return of an image will be described. .

  As shown in FIG. 8A, after the transition to the command reception mode with the first page of the document image G2 such as an electronic book displayed in the virtual image display area 25, the pointer P1 is moved to the virtual image display area by operating the indicator. When the operation is performed so as to move from the virtual image display area 25 beyond the boundary 27d to the outside scene visual recognition area 26 in the diagonally lower right direction with respect to the lower boundary 27d of 25 (the indicator is operated in the direction of arrow E). ), The head-mounted display device HM advances the page of the displayed first-page document image G2 and displays the second-page document image G2, as shown in FIG. 8B. (Page feed command) is executed.

  At this time, the head-mounted display device HM changes the shape of the pointer P1 to an arrow-shaped figure (right arrow-shaped pointer P4) indicating that the page turning process has been performed, and displays it. Further, when the user reaches the limit of the page turning process by repeating the page turning operation (for example, when the page turning operation is performed with the last page displayed), the head-mounted display device The HM changes the pointer to the shape of the prohibition mark and displays it.

  As shown in FIG. 8C, when the indicator is operated by the user so that the pointer P1 moves rightward along the lower boundary 27d (indicator (pointer in the direction of arrow F)). When P1) is moved), it is possible to execute the page turning process in the same manner as described above.

  On the other hand, as shown in FIG. 9A, in the command reception mode, the pointer P1 is moved to the lower side by the operation of the indicator while the second page of the document image G2 such as an electronic book is displayed in the virtual image display area 25. When operated to move to the outside scene visual recognition area 26 from the virtual image display area 25 across the boundary 27d in the diagonally lower left direction with respect to the boundary 27d (when the indicator is operated in the direction of the arrow G), the head As shown in FIG. 9B, the wearable display device HM performs a process of returning the displayed second page document image G2 and displaying the first page document image G2 (page return command). Run.

  At this time, the head-mounted display device HM changes the shape of the pointer P1 to an arrow-shaped figure (left arrow-shaped pointer P5) indicating that the page return process has been performed, and displays it. In addition, when the user reaches the limit of the page return process by repeating the page return operation (for example, when the page return operation is performed with the first page displayed), the head mounted display The device HM changes the pointer to the shape of the prohibition mark and displays it.

  As shown in FIG. 9C, when the pointer is operated by the user so that the pointer P1 moves to the left along the lower boundary 27d (indicator (pointer in the direction of arrow H)). When P1) is moved, the page return process can be executed in the same manner as described above.

  Next, with reference to FIG. 10, an example of a procedure (user interface for executing the rotation process) in the case of executing the image rotation process (rotation command) will be described.

  As shown in FIG. 10A, in the state where the image G3 is displayed in the virtual image display area 25 and the mode is changed to the command reception mode, the pointer P1 is obliquely moved in the upper right direction with respect to the upper right corner boundary 27e by the operation of the indicator. When operated to move from the virtual image display area 25 to the outside scene visual recognition area 26 beyond the boundary 27e in the upper right corner (when the indicator is operated in the direction of the arrow I), the head-mounted display device HM As shown in FIG. 10B, a process (rotation command) for rotating the displayed image G3 by 90 degrees clockwise is executed. At this time, the head-mounted display device HM changes and displays the shape of the pointer P1 to an arrow-shaped figure (pointer P6) indicating that the rotation processing has been performed.

  Note that the pointer P1 moves from the virtual image display area 25 to the outside scene visual recognition area 26 with respect to the remaining three corners of the boundary 27 (lower right corner boundary 27f, upper left corner boundary 27g, lower left corner boundary 27h). Even when the indicator is operated as described above, the rotation process (the process of rotating the displayed image G3 by 90 degrees clockwise) can be executed in the same manner as described above.

  In the above description, the image G3 is rotated 90 degrees clockwise by one rotation operation. However, the image G3 is merely an example, and the rotation direction and the rotation angle are arbitrary. That is, the rotation direction may be counterclockwise, or may be rotated by 45 degrees, for example, by one rotation operation. In addition, the rotation direction and the rotation angle may be individually set according to each corner of the boundary 27 (upper right corner boundary 27e, lower right corner boundary 27f, upper left corner boundary 27g, lower left corner boundary 27h). Is possible.

  Next, with reference to FIG. 11 and FIG. 12, the procedure (user interface for enlargement / reduction process + rotation process) in the case of simultaneously executing the enlargement / reduction process (enlargement / reduction instruction) and rotation process (rotation instruction) of the image An example will be described.

  As shown in FIG. 11A, in the state where the image G4 is displayed in the virtual image display area 25 and the mode is changed to the command reception mode, the pointer P touches the right boundary 27a while rotating right by the operation of the indicator. Is operated (when the movement locus of the indicator is operated to draw a spiral as indicated by an arrow J), the head-mounted display device HM is displayed as shown in FIG. A process of rotating the image G4 clockwise while enlarging the image G4 (enlargement instruction + rotation instruction) is executed. At this time, the head-mounted display device HM changes and displays the shape of the pointer P into a spiral figure (pointer P7) indicating that the enlargement / rotation processing has been performed.

  As a method for controlling the enlargement ratio of the image, for example, the control is performed in consideration of the speed at which the pointer P contacts the right boundary 27a, the number of times of contact, the contact angle with the boundary 27a, and the like. In addition, as a method for controlling the rotation angle of the image, for example, the rotation amount is controlled by the degree of rotation of the indicator in the portion beyond the right boundary 27a.

  On the other hand, as shown in FIG. 12 (a), the image G4 is displayed in the virtual image display area 25, and the pointer P touches the right boundary 27a while the pointer P is rotated counterclockwise by the operation of the indicator in the state of transition to the command reception mode. When the head-mounted display device HM is operated as shown in FIG. 12B, the head-mounted display device HM, as shown in FIG. A process of rotating the displayed image G4 counterclockwise while reducing it (reduction command + rotation command) is executed. At this time, the head-mounted display device HM changes the shape of the pointer P to a spiral figure (pointer P8) indicating that the reduction rotation processing has been performed, and displays it.

  As a method for controlling the reduction ratio of the image, for example, the control is performed in consideration of the speed at which the pointer P contacts the right boundary 27a, the number of times of contact, the contact angle with the boundary 27a, and the like. As a method for controlling the rotation angle of the image, for example, the rotation amount is controlled by the degree of rotation of the pointer P (indicator) after touching the right boundary 27a.

  In addition, when the user reaches the limit of the enlargement process or the reduction process by repeating the enlargement rotation operation and the reduction rotation operation (when it becomes impossible to further enlarge or reduce), the head-mounted display device The HM changes the pointer to the shape of the prohibition mark and displays it.

  As described above, according to the first embodiment, the user operates the indicator so as to move the pointer P to the vicinity of the boundary 27 of the virtual image display area 25, and is assigned in advance to the vicinity of the boundary 27. The predetermined command (processing) can be executed. That is, since the user can execute a predetermined command by a simple (simple) and intuitive operation without performing a complicated operation, it is easy to use. In addition, since it can be performed by a simple operation, it is possible to provide a user interface that operates stably with few erroneous recognitions.

  Note that instructions for performing the enlargement / reduction processing, page movement processing, rotation processing, and the like described in this embodiment may be assigned in combination. For example, when an operation is performed on the left and right boundaries 27b and 27a of the virtual image display area 25 while one image is displayed in the virtual image display area 25, an enlargement / reduction process (enlargement / reduction command) is executed, When an operation is performed on the borders 27c and 27d, a page movement process (page movement command) is executed, and when an operation is performed on the four corner boundaries 27e to 27h, a rotation process (rotation command) is executed. It is also possible to do.

  In addition, the operation method of the indicator for executing the various commands described in the present embodiment (the movement of the pointer P for executing the various commands) is an example, and is not limited thereto.

  Moreover, the allocation positions of various instructions in this embodiment are merely examples, and the present invention is not limited to this. For example, a page movement command may be assigned to the left and right boundaries 27b and 27a of the virtual image display area 25, and an enlargement / reduction command may be assigned to the upper and lower boundaries 27c and 27d.

  It is also possible to assign instructions other than the instructions described in this embodiment. For example, when a movie is displayed in the virtual image display area 25, an instruction for adjusting the volume is assigned to the right boundary 27a, and an instruction for executing fast forward / rewind of the video is assigned to the lower boundary 27d. It is possible to assign an instruction according to the content. In other words, different commands may be assigned to the vicinity of each boundary 27 according to the content.

  In the present embodiment, the description has been given of assigning different commands to the vicinity of the upper, lower, left, and right boundaries 27 (27a to 27d) of the virtual image display area 25. However, the same command can also be assigned. For example, even in the vicinity of the right boundary 27a, a different command can be assigned to each position (neighboring position) such as an upper part and a lower part of the boundary 27a.

  In the present embodiment, the case where the size of the virtual image display area 25 is smaller than the size of the outside scene visual recognition area 26 is illustrated, but the present invention is not limited to this. When the virtual image display area 25 and the outside scene visual recognition area 26 have the same size (in other words, when the virtual image display area 25 has the same size as the visual field 21), or the virtual image display area 25 is larger than the external scene visual recognition area 26 The present invention can also be applied to cases (in other words, the virtual image display area 25 is larger than the field of view 21). In the latter case, the “predetermined display area” in the claims is the edge of the outside scene visual recognition area 26 (the edge of the visual field 21), and the “display peripheral area” is an area outside the external scene visual recognition area 26, that is, outside the visual field 21 It becomes the area of.

[Second Embodiment]
Next, a second embodiment of the present invention will be described with reference to FIGS. In the first embodiment, the image display unit 40 that allows the user to visually recognize a virtual image, the control unit 30 that controls the image display unit 40, and the like are mounted in one housing (inside the device), and the head-mounted display device. In this embodiment, the head-mounted display device HM is configured by realizing and connecting these as separate devices. Hereinafter, differences from the first embodiment will be described. In the present embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and detailed description thereof is omitted. Moreover, the modification applied about the component similar to 1st Embodiment is applied similarly about this embodiment.

  FIG. 13 is an explanatory diagram showing an external configuration of a head-mounted display device HM according to the second embodiment. The head-mounted display device HM includes an image display device 300 that allows the user to visually recognize a virtual image while being mounted on the user's head, and a controller 200 (control device) that controls the image display device 300. ing. The display device in the claims has the controller 200 and the image display device 300 as main components.

  The image display device 300 displays an image in addition to the right holding unit 11, the right display driving unit 12, the left holding unit 13, the left display driving unit 14, the right optical image display unit 15, the left optical image display unit 16, and the sensor 17. A connection part 310 for connecting the apparatus 300 to the controller 200 is provided.

  The connection unit 310 includes a main body cord 311 connected to the controller 200, a right cord 312 in which the main body cord 311 branches in two, a left cord 313, and a connecting member 314 provided at a branch point. . The right cord 312 is inserted into the casing of the right holding unit 11 from the distal end AP in the extending direction of the right holding unit 11 and is connected to the right display driving unit 12. Similarly, the left cord 313 is inserted into the housing of the left holding unit 13 from the distal end AP in the extending direction of the left holding unit 13 and is connected to the left display driving unit 14.

  The image display device 300 and the controller 200 transmit various signals via the connection unit 310. Each end of the main body cord 311 opposite to the connecting member 314 and the controller 200 are each provided with a connector (not shown) that fits with each other. The connector of the main body cord 311 and the connector of the controller 200 are connected to each other. The controller 200 and the image display apparatus 300 are connected to or disconnected from each other by the fitting / releasing of. For the main body cord 311, the right cord 312, and the left cord 313, for example, a metal cable or an optical fiber can be used.

  The controller 200 is an apparatus that is connected to the image display apparatus 300 via the connection unit 310 and controls image display by the image display apparatus 300. The controller 200 includes a lighting unit 201, a touch pad 202, an operation key group 203, and a power switch 204.

  The lighting unit 201 notifies the operation state of the head-mounted display device HM according to the light emission state. The operating state of the head-mounted display device HM is, for example, an ON / OFF state of a power source. For example, an LED (Light Emitting Diode) can be used as the lighting unit 201.

  The touch pad 202 detects a user's finger operation on the operation surface and outputs a signal corresponding to the detected content. For example, a pointing (position designation) operation, a click operation, a drag operation, a flick operation, and the like by the user can be accepted.

  The touch pad 202 of this example also functions as an indicator of the first embodiment. That is, the pointer is displayed in the virtual image display area 25 by operating the touch pad 202, and the pointer is moved based on the operation on the operation surface, thereby displaying the above-described menu screen, enlargement / reduction processing of images, etc., page movement Various commands such as processing and rotation processing can be executed.

  The operation key group 203 includes various operation keys such as a direction key, a determination key, and a back key. Each operation key, when pressed by the user, outputs a signal indicating a process associated with the pressed operation key in advance. The power switch 204 switches the power ON / OFF of the head-mounted display device HM by detecting a slide operation of the switch.

  FIG. 14 is a block diagram functionally showing the configuration of the head-mounted display device HM according to the second embodiment. As illustrated in FIG. 14, the controller 200 includes an operation unit 211 in addition to the control unit 30, the power supply unit 71, and the interface 72. The operation unit 211 includes a touch pad 202, operation key group 203 (direction key, determination key, back key) and power switch 204.

  In addition, the CPU 50 of the control unit 30 reads out and executes a computer program from the storage unit 60, whereby an operating system (OS) 51, an image processing unit 52, a display control unit 53, an image analysis unit 54, and an instruction execution unit 55. In addition to the voice processing unit 56, it also functions as the operation control unit 212.

  The operation control unit 212 receives a signal output from the operation unit 211 in response to a user operation on the operation unit 211, and causes the image processing unit 52, the display control unit 53, and the like to perform an operation according to the signal. For example, a signal indicating that the user has touched the touch pad 202 is received (detected) from the operation unit 211, and the pointer P is displayed on the image processing unit 52, the display control unit 53, or the like. That is, the operation control unit 212 of the present embodiment corresponds to a pointer detection unit in the claims.

  The image display device 300 includes the right display driving unit 12, the left display driving unit 14, the right light guide plate 86 as the right optical image display unit 15, and the left light guide plate 96 as the left optical image display unit 16. And a sensor 17, a right earphone 18, and a left earphone 19. Since the configuration of each part is the same as that of the first embodiment, the description thereof is omitted.

  Thus, the present invention can also be applied to the case where the head-mounted display device HM includes the image display device 300 that allows the user to visually recognize a virtual image and the controller 200 that controls the image display device 300. .

  In this embodiment, the controller 200 and the image display device 300 are illustrated as being connected by wire through the connection unit 310. However, the controller 200 and the image display device 300 may be connected to each other via a wireless LAN, infrared communication, or Bluetooth (registration). (Trademark) or the like.

  Further, the function of the controller 200 of the present embodiment may be realized by a personal computer (PC), a personal digital assistant (PAD, mobile phone, watch-type mobile terminal) or the like.

  In the above-described examples (first embodiment and second embodiment), the binocular head-mounted display device HM (optical transmission type head-mounted display device) is exemplified as the display device. The present invention is also applicable to other types of head mounted display devices such as a video transmission type, a non-transmission type, and a monocular type. In the above embodiment, the eyeglass-type head-mounted display device HM that covers the front of the eye is illustrated, but the present invention is not limited to this, and the type that does not completely cover the front of the eye (the front of the eye). The present invention can also be applied to a head-mounted display device of a type that partially covers). Furthermore, the present invention is not limited to a head-mounted display device, and can be applied to other information processing devices such as a head-up display, a personal computer, a projector, a television, and a portable information terminal.

  In the above embodiment, the image light generation unit includes a backlight control unit (right backlight control unit 81 and left backlight control unit 91), a backlight (right backlight 82 and left backlight 92), an LCD Although the control unit (the right LCD control unit 83 and the left LCD control unit 93) and the LCD (the right LCD 84 and the left LCD 94) are configured, this mode is merely an example. The image light generation unit may include a configuration unit for realizing another method together with or in place of these configuration units.

  For example, the image light generation unit may include an organic EL (Organic Electro-Luminescence) display and an organic EL control unit. Further, for example, the image light generation unit may use LCOS (Liquid crystal on silicon, LCoS is a registered trademark), a digital micromirror device, or the like instead of the LCD. Further, for example, the present invention can be applied to a laser retinal projection type head-mounted display device. In the case of the laser retinal projection type, the “image light emitting area in the image light generation unit” can be defined as an image area recognized by the user's eyes. Further, the optical system of the head-mounted display device may be configured to allow external light to pass through the optical member and enter the user's eyes together with the image light, and may cover a part of the user's eyes. The optical member may be scanning type image light such as a laser, and the optical member is not limited to an object that guides the image light inside the member, but only has a function of refracting or reflecting the image light in the direction of the user's eyes. You may do it.

  In this embodiment, the projection optical system (the right projection optical system 85 and the left projection optical system 95) that projects the image light emitted from the image light generation unit is applied as the optical system. It is an example. For example, a scanning optical system composed of a MEMS mirror may be employed as the optical system, and a signal light modulation unit (signal light forming unit) that emits signal light to the scanning optical system may be provided. In this case, the signal light formed and emitted by the signal light modulation unit is incident on a scanning optical system (MEMS mirror) that is a scanning unit. The scanning optical system emits signal light as scanning light toward a light guide having a half mirror layer, and the scanning light is scanned on the surface of the half mirror layer to form a virtual image by image light. The user can recognize the image by grasping with the eyes. That is, even when a scanning optical system is used, the user recognizes an image such as a pointer by the above-described method, and the execution of an instruction assigned near the boundary 27 of the virtual image display area 25 is controlled by the movement of the pointer. can do.

  Moreover, it is possible to provide each function (each process) of the head-mounted display device HM shown in the above embodiment as a program. Further, the program can be provided by being stored in various recording media (CD-ROM, flash memory, etc.). That is, a program for causing a computer to function as each component of the head-mounted display device HM and a recording medium on which the program is recorded are also included in the scope of the right of the present invention.

  Further, regardless of the above-described embodiments, the device configuration and processing steps of the head-mounted display device HM can be appropriately changed without departing from the gist of the present invention.

  DESCRIPTION OF SYMBOLS 15 ... Right optical image display part 16 ... Left optical image display part 17 ... Sensor 25 ... Virtual image display area 26 ... Outside view visual recognition area 27, 27a-27h ... Boundary 30 ... Control part 40 ... Image display part 52 ... Image processing part 53 ... Display control unit 54 ... Image analysis unit 55 ... Command execution unit 81 ... Right backlight control unit 82 ... Right backlight 83 ... Right LCD control unit 84 ... Right LCD 86 ... Right light guide plate 91 ... Left backlight control unit 92 ... Left Backlight 93 ... Left LCD controller 94 ... Left LCD 96 ... Left light guide plate 200 ... Controller 300 ... Image display device P ... Pointer HM ... Head-mounted display device

Claims (9)

A pointer display unit that displays a pointer that operates in conjunction with an indicator operated by the user in a predetermined display area;
A command execution unit that executes an image scaling process when the pointer is operated so that the pointer moves from the predetermined display area to a display peripheral area that is an area other than the predetermined display area; ,
The pointer display unit displays the pointer in a first shape until the enlargement / reduction processing is performed, and after the enlargement / reduction processing is performed , the pointer display unit has a second shape different from the first shape. display device and displaying. As a result of performing the enlargement process of the image as the enlargement / reduction process, the pointer display unit is in a state where the image cannot be enlarged any more, or as a result of performing the reduction process of the image as the enlargement / reduction process, 2. The pointer according to claim 1, wherein the pointer is changed to a third shape that is different from both the first shape and the second shape after the image is in a state where the image cannot be further reduced . Display device.   The instruction execution unit sets an image scaling ratio according to at least one of a speed at which the pointer contacts the boundary between the predetermined display area and the display peripheral area, the number of times of contact, and a contact angle with the boundary. The display device according to claim 1, wherein the display device is controlled. The instruction execution unit executes the enlargement / reduction process of the image when the pointer moves to the vicinity of the part to which the instruction is assigned in the boundary between the predetermined display area and the display peripheral area,
The pointer display unit includes the pointer when the pointer moves in the vicinity of a portion to which an instruction is assigned in the boundary, and the pointer in the vicinity of a portion in the boundary to which an instruction is not assigned. The display device according to claim 1, wherein the pointer when it is moved is displayed in a different shape. The display device according to any one of claims 1 to 4,
The display device is a transmissive display device that allows a user to visually recognize an image as a virtual image,
A display device characterized in that a user detects the indicator operated in front of the outside of the display device, which is the opposite side to the user side of the transmissive display device. The display device according to any one of claims 1 to 4,
The display device is a transmissive display device that allows a user to visually recognize an image as a virtual image,
The transmissive display device includes:
An image light generation unit that generates and emits image light representing an image using image data to be displayed; and
An optical member for guiding the emitted image light to the eyes of the user,
A display device characterized in that light of an outside scene passes through the optical member and enters the user's eyes together with the image light. A transmissive head-mounted display device that allows a user to visually recognize an image as a virtual image,
A pointer display unit that displays a pointer that operates in conjunction with an indicator operated by the user in a predetermined display area;
A command execution unit that executes an image scaling process when the pointer is operated so that the pointer moves from the predetermined display area to a display peripheral area that is an area other than the predetermined display area; ,
The pointer display unit displays the pointer in a first shape until the enlargement / reduction processing is performed, and after the enlargement / reduction processing is performed , the pointer display unit has a second shape different from the first shape. head-mounted display device and displaying.   The head-mounted display device according to claim 7, wherein the indicator is detected by imaging the front of the user in a state where the head-mounted display device is mounted. Display device
A pointer display step for displaying a pointer that operates in conjunction with an indicator operated by the user in a predetermined display area;
A command execution step for executing an image scaling process when the pointer is operated so that the pointer moves from the predetermined display area to a display peripheral area that is an area other than the predetermined display area; And
In the pointer display step, the pointer is displayed in the first shape until the enlargement / reduction processing is performed, and after the enlargement / reduction processing is performed, the pointer is displayed in a second shape different from the first shape. A method for controlling a display device, characterized by displaying.

Images