JP3214158B2 - Key input processing device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a key input processing apparatus provided with a display means for forming and displaying a display image in front of an operator's eyes while being mounted on the operator.

[0002]

2. Description of the Related Art Conventionally, as a processing apparatus, for example, an engineering workstation (hereinafter, EWS: Engi)
neering Work Station), a personal computer (hereinafter abbreviated as a personal computer), a word processor (a word processor), and the like.

[0003] These processing devices are usually provided with a keyboard as an input device and a display device as an output device. Processing corresponding to key commands input from the keyboard is performed in the processing device. The result is output to the display device.

A typical display device in a processing device includes a CRT (Cathode Ray Tube) display device and a liquid crystal display device (hereinafter, LCD: Liquid Crystal Diode).
splay).

A CRT display device is generally used as a display device in a desktop type EWS, a personal computer, a word processor, and the like. A large-screen display device can be obtained relatively inexpensively. When a large-screen display device is obtained, the size of the CRT display device itself becomes large, and the installation place is limited.

On the other hand, LCDs have low power consumption and are lightweight.
Because it is thin, it is widely used as a display device for laptops and notebooks, such as personal computers and word processors. In recent years, large screens and high definition have been achieved, and it is also used as a display device for EWS. ing.

However, in general, it is difficult to increase the screen size of the LCD, and when the screen size increases, the cost increases.

Therefore, HMDs have attracted attention as display devices capable of increasing the screen size regardless of the installation location.

[0009] The HMD displays images on two liquid crystal displays placed in front of the eyes, and sets a convergence angle and a focal length to predetermined values, so that both eyes can be used like an eye mirror or a binocular mirror. In this case, different images are given to each other, and the images are displayed as if a large-screen display device is viewed by a small device. For example, the left and right liquid crystal displays are tilted by 1.5 degrees with respect to the image forming surface. Given a convergence angle of 3 degrees, considering that a person with a 65 mm distance between eyes (Japanese average) gazes about 1 m ahead, a liquid crystal display of about 2 inches type and a focal length of about 20 mm Assuming that an angle of view of 30 degrees is realized by combining with a lens, an optical system having a display screen of about 20 inches 1 m ahead can be obtained.

[0010] Since the HMD has such advantages, it can be used for movie watching, medical treatment by pseudo 3D display (stereoscopic display),
Promising uses in the fields of CAD and architecture.

[0011]

A CRT as an output device
In a conventional device that includes a display device such as an LCD and an LCD, and also has a keyboard as an input device, key input required for data processing can be performed from the keyboard, but a data processing device using an HMD as a display device was considered. In such a case, since the HMD is configured to be used by being mounted on the operator's head, the following problems may occur.

That is, in the above-mentioned HMD, since the display device is mounted on the head, an operator other than a person who can perform a complete blind touch input requires:
A keyboard or the like, which is a general input device, cannot be used. Therefore, when the HMD is mounted, only a pointing device that can be used by a simple operation such as a joystick can be operated.

It is also conceivable to provide a key input unit on the HMD itself. However, it is desired that the HMD be small and lightweight for the purpose of being mounted on the head, and it is not preferable to provide various types of keys.

An object of the present invention is to make it possible to easily execute key input processing in a display image formed and displayed in front of an operator's eye, and a real image in front of the display image is transparently displayed. That is.

[0015]

The means of the present invention are as follows.

Display means mounted on the operator's head, forming and displaying a display image in front of the operator's eyes in the mounted state, and transparently displaying an actual image in front of the display means; Virtual keyboard display means for displaying a virtual keyboard image having a plurality of virtual keys as the display image on the display means in a state where is mounted on the operator's head, and an operator transparently displayed from the front of the display means If the position of the tip of the hand is at the display position of any virtual key in the virtual keyboard image, key input processing means for executing key input processing for the virtual key is provided. Therefore, if the actual tip of the operator's hand transparently displayed from the front of the display device is at the display position of any virtual key in the virtual keyboard image fixedly displayed in the display means, the key input process is executed. can do. If the tip position of the operator's hand transparently displayed from the front of the display means is at the display position of any virtual key in the virtual keyboard image, the virtual key can be identified from other virtual keys. The display state may be changed in advance.

[0017]

[0018]

[0019]

[0020]

[0021]

[0022]

[0023]

An embodiment will be described below with reference to FIGS.

FIGS. 1 to 15 show an embodiment of the data processing apparatus according to the present invention.

First, the configuration will be described. 1 and 2 are external views of a data processing device of the present invention. FIG. 1 is a diagram of the data processing device of the present embodiment viewed obliquely from the front.
FIG. 2 is a diagram of the data processing device of the present embodiment viewed obliquely from behind.

The data processing device 1 according to the present embodiment
Since it is built in a single display device by MD and has no key input means such as a keyboard, by displaying a virtual keyboard on the HMD,
This enables virtual key input.

In FIG. 1 and FIG.
Incorporates an optical system display unit 2 for generating various data processed in data processing as a display image, and various units for combining and displaying an image generated by the optical system display unit 2 and an outside scenery. Equipment case 3,
It comprises an optical system display unit 2 and a device case 3, and a telescopic headband 4 attached to the head.

As shown in FIG. 1, the device case 3 is provided with a rod antenna 5, a power on / off switch 6, a designation key 7, a menu switching down instruction key 8, and a menu switching up instruction key 9. As shown in FIG. 2, the viewfinders 10R and 10L are formed.
As shown in FIG. 1, the support part 11 and the earphones 12 </ b> R and 12 </ b> L are formed on the headband 4.
As shown in FIG. 2, a mounting detection sensor 13 is provided.

The rod antenna 5 is a telescoping antenna for converting various data processed in the data processing device 1 into radio waves and transmitting the data. The power on / off switch 6 includes a power supply unit (not shown). ) Is a switch for selecting whether or not to supply the power supply voltage supplied from the data processing apparatus 1 to the data processing apparatus 1. The designation key 7 is used to confirm the operator who uses the data processing apparatus 1 when the data processing apparatus 1 is used. Switch to start. The antenna to be used is not limited to the telescopic rod antenna 5 as in this embodiment, but may be, for example, a helical antenna, and an antenna suitable for the wavelength band of the radio wave to be used can be used.

The menu switching down instruction key 8 and the menu switching up instruction key 9 are key switches for switching menu contents of a menu screen displayed by the optical system display unit 2, and the menu switching down instruction key 8 is operated in order. The menu contents are switched in the direction (the direction in which the frame cursor in the menu screen is moved downward), and the menu switching up instruction key 9 is in the opposite direction to the menu switching down instruction key 8 (the frame cursor in the menu screen is moved upward). (The direction in which the menu is moved).

The viewfinders 10R and 10L are viewing windows provided with a transparent filter, and the support portion 11 supports and fixes the device case 3 and the headband 4.

The earphones 12R and 12L are for listening to error sounds and guidance information, and the wearing detection sensor 13 is a sensor for detecting whether or not the headband 4 is worn on the head of the operator. .

FIG. 3 is a cross-sectional view of the data processing device 1 of the present embodiment as viewed from the lateral direction, and FIG. 4 is a cross-sectional view of the data processing device 1 of the present embodiment as viewed from above.

As shown in FIG. 3, the optical system display unit 2 includes a backlight 21 composed of a side-light type cold-cathode tube (fluorescent lamp) and an LCD 22R, which is an active matrix type color liquid crystal display of about 1 inch in size. , 2
2L and concave lenses 23R and 23L made of a glass or plastic optical lens. The image (light) illuminated by the backlight 21 and transmitted through the LCDs 22R and 22L is spread by the concave lenses 23R and 23L, and the device case is efficiently made. Sent to 3.

As shown in FIGS. 3 and 4, the equipment case 3 includes an upper glass 31 serving as a light receiving portion for receiving an image (light) from the optical system display unit 2 and a glass or plastic optical lens. Convex lens 32R, 32
L, a front glass 33 serving as a light receiving unit for receiving an external scene (external light), and a transmission state and a light shielding state of whether or not the external light received from the front glass 33 is guided into the device case 3. LCD 34 for switching, which is a liquid crystal shutter for switching
A reflective magic mirror 35 formed of a half mirror formed inside the device case 3 of the shutter LCD 34;
Two CCDs provided on the left and right side walls in the equipment case 3
(Charge Coupled Device: image sensor) 36R, 36L
The image (light) obtained from the optical system display unit 2 is reflected by the reflection magic mirror 35 through the glass 31 and the convex lenses 32R and 32L, and is reflected by the operator's eyes ER and ER through the viewfinders 10R and 10L. EL
Reaches Furthermore, when the shutter LCD 34 is in the transmissive state, the image of the surrounding situation seen forward through the front glass 33 is combined with the image from the optical system display unit 2 via the reflective magic mirror 35, and the viewfinders 10R and 10L.
On the other hand, when the shutter LCD 34 is in the light-shielded state, the image of the surrounding state seen forward through the front glass 33 does not reach the operator's eyes ER and EL, and only the image from the optical system display unit 2 is displayed. Is output to the operator.

Here, the two CCDs 36R, 36L provided on the left and right side walls in the equipment case 3 capture images in either direction of the front or rear viewfinders 10R, 10L. It is provided rotatably so as to rotate 180 degrees to the rear and back. When the shutter LCD 34 is in the transmission state, two C
The front scene is imaged by the CDs 36R and 36L, and for example, it is detected whether or not the “hand” is in the front view,
If there is a "hand", processing is performed to determine the position of the "hand" and the distance from the HMD to the "hand" based on the parallax of the two CCDs 36R and 36L. LC for shutter
When D34 is in a light-shielded state, two CCDs 36R,
36L rotates 180 degrees toward the viewfinders 10R and 10L, and captures images of the operator's eyes ER and EL through the viewfinders 10R and 10L. The captured eye image is
This is compared with the image data of the eyes of each operator stored in the data memory described later. Incidentally, the eye data at the time of the comparison processing includes, for example, the ratio of the corneal (black eye) portion and the conjunctival (white eye) portion based on the pigment distribution, and the shape of the peripheral portion of the eye (shape of eyebrows, eyelid shape, eye shape, etc.). Overall shape),
Alternatively, if the location of the blood vessel can be specified and it is difficult to cause redness, it is conceivable to use the blood vessel pattern of the eye.

FIG. 5 is a block diagram showing the overall configuration of the data processing apparatus 1 of the present embodiment.

The data processing device 1 of the present embodiment has a CPU 4
1, ROM 42, data memory 43, angle sensor 44,
An orientation sensor 45, an infrared sensor 46, an input unit 47, a display unit 48, a display drive unit 49, and a wireless transmission control unit 50 are provided.
Further, it includes a mounting detection sensor 13 and CCDs 36R and 36L. In FIG. 4, B represents a CPU 41 and a ROM.
42, data memory 43, angle sensor 44, direction sensor 45, infrared sensor 46, input unit 47, display drive unit 4.
9, wireless transmission control unit 50, mounting detection sensor 13, CCD
Buses serving as signal transmission paths for connecting 36R and 36L, 101 are a printer, and 102 is an external storage device.

The CPU 41 outputs various control signals for controlling each section in the data processing device 1 to each circuit via the bus B. The CPU 41 also outputs a control signal based on input information from the input section 47. The display control unit 49 controls the display drive unit 49 to perform a predetermined display on the display unit 48, and the wireless transmission control unit 5.
0 to convert various data processed in the data processing device 1 into radio waves,
2 transmits the necessary data.

The ROM 42 is a semiconductor memory for storing programs and data used in the data processing device 1.

The data memory 43 stores various data used in the processing executed by the CPU 41, that is, display data, image data of "hands", eye data for each operator, and a work area in data processing. Is a semiconductor memory having a work area. The display data is display data of various menu screens and the like displayed by the optical system display unit 2.
The “hand” image data is virtual “hand” data displayed by the optical system display unit 2 based on the “hand” image detected in the forward view of the HMD, and eye data for each operator. Are the eyes ER, EL of the regular operator
, That is, image data serving as a comparison reference for performing operator recognition.

The angle sensor 44 is, for example, used in a navigation system, a gyro system, or the like.
This sensor detects how much the HMD, which is the data processing device 1, is tilted from a horizontal state by detecting "sway". In the present embodiment, as shown in FIG. 6, a position rotated by a predetermined angle clockwise from the vertical direction in FIG. 6 is set as a reference point position (0-degree position). It is set to face upward.

The azimuth sensor 45 is a sensor for detecting how many times the HMD, which is a data processing device, has rotated left or right from the reference position. In the present embodiment, as shown in FIG. 7, the azimuth of “north” is set to the reference point position (N in FIG. 7), and the number of clockwise rotations in FIG. 7 is detected. Has become.

The infrared sensor 46 includes CCDs 36R, 36
In order to prevent erroneous recognition, for example, when the image of the "hand" detected by L is a toy "hand", the "hand" in the forward view is detected by the CCDs 36R and 36L. After the position of the “hand” and the distance from the HMD to the “hand” are obtained, the CCD image and the “hand” are obtained.
If the thermal data is detected, it is checked whether there is a thermal reaction at the location where the image data of the "hand" is located based on the position and the distance of the "hand". When the CPU 41 determines that there is a "hand" and no thermal reaction is detected, the CPU 41 determines that the CCDs 36R and 36L have erroneously recognized.

The input unit 47 is a virtual input unit provided to enable key input in the data processing device 1. This is because the data processing device 1 of the present embodiment includes only a display unit and does not particularly have an input unit such as a keyboard, so that a keyboard is displayed on the display unit 48 as display data in the data memory 43. By superimposing and displaying the "hands" which are transparently projected, the positional relationship is clarified to enable key input.

As shown in FIG. 3, the display unit 48 includes an optical unit (backlight 21, LCD 22R, 22L, concave lens 23R) in the optical system display unit 2 for displaying two images respectively corresponding to the left and right eyes of the operator. , 23
L) and the optical unit (convex lens 3) in the device case 3.
2R, 32L, reflection magic mirror 35, etc.) and fixed at the point where the two images are combined, the images viewed from the left and right viewfinders 10R, 10L are combined and displayed in a natural state. .

The display driving section 49 includes the LCDs 22R and 22L.
Is a display driving unit that drives the LCDs 22R and 22L to display images according to
Image data generated based on the process of U41
It converts the analog video signals into analog video signals (analog RGB signals) of predetermined bits corresponding to the number of pixels of 22R and 22L, and outputs the analog video signals to the LCDs 22R and 22L.

The wireless transmission control unit 50 includes the data processing device 1
It converts various data processed in the system into radio waves and transmits the radio waves. This is because the data processing device 1 of the present embodiment is composed of only a display unit and cannot output processed data by itself. For example, in the case of performing printout, desired data to be printed out is required. Is modulated by the wireless transmission control unit 49 by frequency modulation, amplitude modulation, or the like, and this data is output to an external printer 101 as a radio wave. Similarly, when important data is stored, desired data to be stored is stored. The data is converted into radio waves and output to the external storage device 102. In this case, data can be transmitted / received to / from the same data processing apparatus 1 as in the present embodiment, and transmission radio waves from one host station can be transmitted to a plurality of data processing apparatuses 1 having the same configuration as in the present embodiment. By receiving the information, a common image can be viewed, and for example, it can be used for a conference or the like.

Next, the operation of this embodiment will be described.

FIG. 8 is a flowchart of an operator recognition process when the data processing apparatus of this embodiment is started.

Hereinafter, an operator recognition process by the data processing device 1 will be described with reference to FIG. FIG.
The program corresponding to the processing operation of the CPU 41 shown in
It is stored in the ROM 42.

First, the data processing device 1 in the present embodiment
Is mounted on the operator and the power on / off switch 6 is turned on, a menu screen is displayed on the display unit 48 of the HMD (step S1). In this state, the menu switching down instruction key 8 or the menu switching up instruction is issued. Key 9
Is checked (step S2), and if the menu switching down instruction key 8 or the menu switching up instruction key 9 is pressed, the menu contents are switched based on the switching instruction (step S3).

On the other hand, in the process of step S2,
If the menu switching down instruction key 8 or the menu switching up instruction key 9 is not pressed, then the designation key 7
Is checked (step S4), and if the designation key is not pressed, the processing from step S2 is repeated.

In the process of step S4, the designation key 7
Is pressed, the mounting detection sensor 13 checks whether the HMD is mounted (Step S).
5) If not mounted, step S12 described later
Proceed to error processing of.

In step S5, when the mounting of the HMD is confirmed, the CCDs 36R and 36L are inverted, and
The direction is changed in the directions of the viewfinders 10R and 10L (step S6).

The images of the eyes ER and EL of the operator are picked up by the CCDs 36R and 36L (step S7), and the images of the eyes are analyzed to generate image data of the left and right eyes (step S8).

Next, in step S8, it is checked whether or not the image data of the eye has been properly generated (step S8).
9) If not generated, step S described later
When the processing proceeds to error processing 12 and the generated eye image data is compared with the eye data for each operator in the data memory 43 by the CPU 41, the registered eye ER and EL data ( For example, it is checked whether or not they match the above-described black eye portion, white eye portion, and pigment ratio (step S10).

As a result of the check in step S10, the registered eye data and the eye E of the operator who wears the HMD.
If the data of R and EL match, the CPU 41 determines that the data is used by a regular operator, and switches to an initial menu screen on which each function processing is performed according to the designated menu (step S11), while checking in step S10. As a result, when the registered eye data and the data of the eyes ER and EL of the operator wearing the HMD do not match, C
The PU 41 determines that the operator currently wearing the HMD is an unauthorized use by an operator other than the registrant, and a predetermined error process is executed (step S12).

When the processing of step S11 or step S12 is completed, the CCDs 36R and 36L are inverted and C
The directions of the CDs 36R and 36L are changed again in the forward direction of the HMD (step S13), and the operator recognition processing ends.

As described above, if it is determined in the operator recognition process that the use is made by a regular operator through the processing of steps S1 to S13, an initial menu screen as shown in FIG. 9A is displayed. By the way,
In the example shown in FIG. 9A, the selection item is "data input /
Items of "edit", "data recognition input", "recording", and "playback" are displayed, and a state in which a frame cursor is located in the item of "data input / edit" is shown.
When the menu switching down instruction key 8 is pressed, the frame cursor moves downward in the figure, and when the menu switching up instruction key 9 is pressed, the frame cursor moves upward in the figure to highlight the target item.

Here, as shown in FIG. 9A, when the designation key 7 is pressed while the frame cursor is at the item of "data input / edit", the function of "data input / edit" is selected. A data menu screen as shown in FIG. 9B is displayed. In FIG. 9B, A is a layout window for indicating or designating the layout of data to be displayed, and B is an operation for displaying an item selected on the initial menu screen of FIG. 9A. This is an item display window, from which the current operation content can be confirmed. C is
Indicates the file name of the document or table data to be edited,
A file name window for designating a file name of a document or table data to be edited, and D is a keyboard designation window for designating a virtual keyboard.

Next, the data input / edit processing will be described with reference to the flowcharts shown in FIGS. FIG. 10 is a flowchart showing the data input / edit processing.

In the data menu screen shown in FIG. 9B, at the time of data input / edit processing, the data of the virtual keyboard is read from the data memory 43 and the processing does not proceed to the next processing. For this reason, first, it is checked whether the virtual keyboard is displayed by designation (step S).
21) If the virtual keyboard is displayed, the process proceeds to the next step. On the other hand, if the virtual keyboard is not displayed, a keyboard designation process described later is executed (step S22), and the virtual keyboard is displayed.

On this virtual keyboard, a "read key"
It is checked whether a "read key" has been selected (step S23).

When the "read key" is selected in the process of step S23, a list of storage file names is displayed in a window (step S24), and the "cursor key" on the virtual keyboard is operated in this state (step S24). In step S25, the cursor position is changed, and the display content is updated to the file name at the changed cursor position (step S26).

Next, the presence or absence of the window input designation is checked (step S27).
The file data corresponding to the designated position is read (step S28), the previous display is erased, and the layout window is displayed again (step S29).

Here, the presence or absence of the window input designation is checked again (step S30), and if there is an input designation, a file data arrangement process is performed (step S3).
1) The arranged file data is displayed, and the file name window is deleted (step S3).
2), the data input / edit processing ends.

On the other hand, in the processing of the step S27 and the processing of the step S30, if there is no input designation of the window, it is checked whether or not the "clear key" on the virtual keyboard is pressed (step S33). If is not pressed, the processing from step S24 is repeatedly executed. If the "clear key" is pressed, the data input / edit processing ends.

If the "read key" is not selected in the process of step S23, it is checked whether the "write key" prepared on the virtual keyboard is selected in the same manner as the "read key". (Step S
34).

If the "write key" is not selected in the process of step S34, the process proceeds to another process. If the "write key" is selected, a list of storage file names is displayed on a window. (Step S35), the presence or absence of input designation is checked (Step S36).

If there is an input designation, it is checked whether display data has been designated (step S3).
7) When the designation of the display data is confirmed, the contents of the designated display data are written to the corresponding file (step S38), and the data input / edit processing ends.

If there is no input designation in the process of step S36, it is determined whether or not a new file is to be created (step S39). If a newly created file is input, a new file is created. (Step S4
0), the processing from step S37 is executed.

If the input is not a newly created file, it is determined that the file name is to be updated. When the "cursor key" is operated (step S41), the cursor position is changed, and the file at the changed cursor position is changed. The display content is updated to the name (step S42), and the processing from step S35 is executed.

If there is no operation of the "cursor key" in the process of step S41, it is checked whether the "clear key" on the virtual keyboard is pressed (step S4).
3) If the "clear key" has not been pressed, the processing from step S35 is repeatedly executed. If the "clear key" has been pressed, the data input / edit processing ends.

FIG. 11 is a flowchart showing the keyboard designation process in FIG.

As shown in FIG. 11, in the above-described keyboard designation process, first, an input designation process described later is executed (step S51), and it is checked whether or not a keyboard window has been designated in the input designation process. (Step S52).

In the process of step S52, if the keyboard designation window is not designated, it is checked whether or not the "clear key" is designated on the virtual keyboard (step S53). If the "clear key" is designated, it is stored. After the cleared keyboard data is set to the initial state (step S54), the processing from step S51 is repeatedly executed. Here, if the "clear key" has not been specified, it is checked whether or not the data menu screen shown in FIG. 9B has been specified (step S55), and if the data menu screen has been specified. Then, the keyboard designation process ends.

If the data menu screen is not specified in the process of step S55,
It is checked whether the fixed flag is on (step S56). Here, when the fixed flag is on, the virtual keyboard is fixedly displayed at the center position of the display unit 48 regardless of the change in the field of view due to the movement of the HMD, and the fixed flag is off. In this case, a virtual keyboard is placed in a predetermined position, just like a real keyboard.
The display mode is such that the keyboard enters or disappears from the field of view due to the movement of the HMD.

That is, when the fixed flag is on,
If it is determined that the virtual keyboard is in the fixed display mode and the processing from step S51 is repeatedly executed, and if the fixed flag is off, the angle sensor 4
4, the current HMD angle is detected, and the direction sensor 45 detects the current HMD direction, and the respective values (angle and direction) are displayed on the display unit 48 (step S57). Then, when the user moves into the area where the keyboard should be displayed by moving the HMD (step S5).
8) The virtual keyboard for the keyboard area is displayed on the display unit 4.
8 (step S59), and when the user is not in the keyboard area, the processing from step S51 is repeatedly executed.

On the other hand, in the process of step S52, when a keyboard designation window is designated, it is checked whether or not a layout window is designated (step S60).

The layout window is specified prior to the specification using the keyboard specification window.
In order to display the virtual keyboard at the front center position no matter where the MD position is, keyboard data is stored in a storage position corresponding to the current angle and direction of the HMD (step S62), and the display of the virtual keyboard is fixed. (Step S63). At this time, the fixed flag is turned on (step S63), the layout position of the virtual keyboard is displayed (step S64), and the processing from step S51 is repeatedly executed.

When the layout window is designated, the data memory 43 corresponding to the layout position
Keyboard data is stored (step S65).
A mode in which the virtual keyboard is displayed only when the MD is directed to that position is entered, and the process proceeds to step S64.

FIG. 12 is a flowchart showing the input designation process in FIG.

As shown in FIG. 12, in the input designation process,
First, the contents of the previous keyboard designation window are cleared (initialized) (step S71).
An image in front of the HMD is captured and captured by 6R and 36L (step S72).

Then, it is checked whether or not the image data of the operator's "hand" exists in the captured image data (step S73). If the image data of the "hand" does not exist, an input designation process is performed. finish.

If the image data of the "hand" exists, the position of the "hand" of the operator with respect to the key display position is obtained (step S74), and there is a thermal reaction by the infrared sensor 46 at the image position of the "hand". It is determined whether or not the thermal reaction is detected (step S75). If no thermal reaction is detected, it is determined to be erroneous recognition, and the input designation process ends.

Next, at the time of the first start-up, the input designation is designated by pressing the designation key 7, but in this state, the input designation is at the designated display position since the input state is from the virtual keyboard. It is checked whether or not this is the case (step S76). If the position is outside the designated display position, the input designation process ends.

In the processing in step S76, the virtual keyboard and the image data of the "hand" overlap, and if the CPU 41 determines that the input is specified at the specified display position, the image data of the "hand" on the virtual keyboard is displayed. The luminance of the key overlapping with is changed (for example, the luminance is increased) (step S77), so that it is possible to know which key the image data of the "hand" touches. In this case, the display state of the transparently projected "hand" and the virtual keyboard is as shown in FIG.

Here, it is determined whether the tip of the "hand" is at least a predetermined distance from the HMD, for example, whether or not the tip of the "hand" is at least 60 cm away from the HMD.
If the position is detected by 6L (step S78) and the leading end position of the "hand" has not reached the predetermined distance, the input designation processing ends.

In the process of step S78,
If it is determined that the tip position of the “hand” has reached a certain distance, a state is possible in which keys can be input to the virtual keyboard, and when a key on the virtual keyboard is pressed over a certain amount,
The corresponding key blinks for a certain period of time.

After the input designation process is completed, in order to use various processing data in the input designation process during the process of calling the input designation process, the key or window content corresponding to the position of the "hand" is used. It is stored in the memory 43 (step S79).

FIG. 13 is a diagram showing an example of an actual display on the HMD of this embodiment.

In FIG. 13, E is a display indicating the azimuth of the HMD detected by the azimuth sensor 45, F is a display indicating the angle of the HMD detected by the angle sensor 44, and G is a predetermined area. A layout window showing how the specified document, table data, and the like are arranged, H is a transparently projected “hand”, I is a virtual keyboard, J is a file name window, and K is the current The open document data L is a surrounding state that can be seen because the HMD is in a transparent state, and in this example, it shows a state where the desk lamp is visible.

As described above, the data processing apparatus 1 of the present embodiment
In (HMD), the surrounding state transmitted by the reflective magic mirror 35 serving as a half mirror is also displayed on the display unit 48 in an overlapping manner.

FIG. 14 is a flowchart showing a display process executed at regular intervals.

In this display processing, first, the angle sensor 44
Then, the angle and direction of the HMD are detected by the direction sensor 45 (step P1), the corresponding area of the display data in the data memory 43 corresponding to the current position of the HMD is searched, and the pointer is moved (step P2).

Then, it is checked whether or not there is data to be displayed in the corresponding area (step P3). If there is display data in the corresponding area, the process proceeds to the next processing, and there is no display data in the corresponding area. In this case, the display processing ends.

If there is display data in the corresponding area, a fixed flag indicating the current display mode is checked (step P3).

If the fixed flag is ON, it is determined that the current display mode is the keyboard fixed display, and the virtual keyboard is displayed at the center of the display unit 48 (step P5).

Then, it is checked whether or not the display data to be displayed in the corresponding area overlaps the display of the virtual keyboard (step P6). If the display data overlaps, only the display area corresponding to the virtual keyboard is displayed in the corresponding area. Move and display the display data.

More specifically, for example, as shown in FIG. 15 (a), there is display data R0, R1, R2 between the left position of the display area pointer indicating the corresponding area and the right position of the display area pointer. As shown in FIG. 15B, the display data is moved upward in FIG. 15 by the amount corresponding to the display of the virtual keyboard, and the virtual keyboard is fixedly displayed (step P7).

On the other hand, if the fixed flag is turned off in the processing of step P4,
If there is no display data that overlaps with the virtual keyboard in the display area in the processing of (3), the data of the corresponding area is displayed as it is (step P8). In this case, the fixed flag is turned on in the process of step P4, and
If the virtual keyboard does not overlap with the virtual keyboard, it is necessary to combine and display the virtual keyboard and the data of the corresponding area.

In the data processing device 1 of this embodiment, H
The display data sequentially displayed based on the position of the MD can be obtained by designating the displayed data by image data of a “hand” in the same manner as inputting from a virtual keyboard (in the case of this embodiment, “ It can be moved to another display position (specified by the operation of pressing with the tip of the hand).

As described above, in this embodiment, the virtual keyboard displayed by the data display means is
The key switch in the designated display data is specified based on the key switch position of the virtual keyboard calculated by the position detecting means and the distance to the key switch calculated by the distance calculating means, and the image of "hand" Key switches to be input can be selectively input from the virtual keyboard by data.

Therefore, even when the HMD is used, virtual keyboard input is possible, and desired data input can be performed without lowering the input ability.

In the above embodiment, the eye data is
For example, when the ratio of the pigment distribution between the iris portion and the iris portion, the shape of the periphery of the eye (shape of eyebrows, eyelid shape, shape of the entire eye, etc.), or the location of blood vessels can be specified, and it is difficult to cause redness. In this case, a blood vessel pattern of an eye was considered, but the present invention is not limited to this. Further, after the shutter LCD 34 is in a light-shielding state and the inside of the device case 3 is darkened, light of a constant illuminance (for example, a fluorescent lamp or the like) is emitted. Irradiation may be used to recognize the operator by detecting the respective ratios of the sphincter of the pupil, the dilated pupil, and the lens body in the pupil of the operator.

[0107]

According to the first aspect of the present invention, the actual tip position of the operator's hand transparently displayed from the front of the display means is displayed in any of the virtual keyboard images fixedly displayed in the display means. If the virtual key is displayed at the display position, the key input processing is executed, so that the key input processing can be easily executed as if the actual keyboard is operated by the operator himself. According to the second aspect of the present invention, a virtual key that has been input-processed by an operator in a virtual keyboard image can be reliably identified from other virtual keys.

[0108]

[0109]

[Brief description of the drawings]

FIG. 1 is a diagram of a data processing apparatus according to a first embodiment as viewed obliquely from the front.

FIG. 2 is a diagram of the data processing device according to the present embodiment as viewed obliquely from behind.

FIG. 3 is a cross-sectional view of the data processing device according to the embodiment as viewed from the side.

FIG. 4 is a cross-sectional view of the data processing device according to the present embodiment as viewed from above.

FIG. 5 is a block diagram illustrating an overall configuration of a data processing device according to the present embodiment.

FIG. 6 is a diagram for explaining angle detection of the data processing device of the present embodiment.

FIG. 7 is a diagram for explaining azimuth detection of the data processing device of the present embodiment.

FIG. 8 is a flowchart of an operator recognition process when the data processing apparatus according to the present embodiment is activated.

FIG. 9 is a diagram showing an initial menu screen and a data menu screen.

FIG. 10 is a flowchart showing data input / edit processing.

FIG. 11 is a flowchart showing a keyboard designation process in FIG.

FIG. 12 is a flowchart illustrating an input designation process in FIG. 11;

FIG. 13 is a diagram illustrating an example of an actual display on the HMD according to the present embodiment.

FIG. 14 is a flowchart showing a display process executed at regular intervals.

FIG. 15 is a diagram illustrating a display example of a virtual keyboard and display data.

[Explanation of symbols]

 Reference Signs List 1 data processing device 2 optical system display unit 3 device case 4 headband 5 rod antenna 6 power on / off switch 7 designation key 8 menu switching down instruction key 9 menu switching up instruction key 10R finder 10L finder 11 support part 12R earphone 12L earphone 13 Mounting Detection Sensor 21 Backlight 22R LCD 22L LCD 23R Concave Lens 23L Concave Lens 31 Upper Glass 32R Convex Lens 32L Convex Lens 33 Front Glass 34 LCD for Shutter 35 Reflection Magic Mirror 36R CCD 36L CCD 41 CPU 42 ROM 43 Data Memory 44 Angle Sensor 45 46 infrared sensor 47 input unit 48 display unit 49 display drive unit 50 wireless transmission control unit 101 printer 102 external storage device

──────────────────────────────────────────────────続 き Continued on the front page (58) Field surveyed (Int. Cl. ⁷ , DB name) G06T 15/00-17/50 G06F 3/00

Claims (2)

(57) [Claims] 1. The operating device in a state where the operating device is attached to an operator.
A display image is formed and displayed in front of the
Display means for transparently displaying the real image in front of the camera, and the display means being mounted on an operator.
A virtual keyboard image with multiple virtual keys
Display as the display image on the display means.
Virtual keyboard display means, and an operator which is transparently displayed from the front of the display means.
The position of the tip of the data hand
If the virtual key is in the display position,
And a key input processing means for executing key input processing. 2. A transparent display from the front of said display means.
The position of the tip of the operator's hand to be
If any of the virtual keys in the
Display status so that virtual keys can be distinguished from other virtual keys
The key input processing according to claim 1, wherein the key input processing is changed.
Equipment.