JP2008009490A - Information input device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an information input device for easily and accurately inputting information to a virtually displayed video in an information input device using an HMD. <P>SOLUTION: This information input device is provided with a display means mounted on the head or the face for displaying a video, and for making an external world see-through; a virtual display frame for virtually displaying a video; and a pointer for inputting information to a virtually displayed video. This information input device is provided with a position detection means for detecting the position of the virtual display frame and the position of a pointer; a composition means for combining the orbit of the position of the pointer detected by the position detection means with the video; and a display position control means for controlling the position where the video is virtually displayed according to the position of the virtual display frame detected by the position detection means. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an information input device, and more particularly to an information input device using a head-mounted video display device.

  2. Description of the Related Art Conventionally, in the field of VR (Virtual Reality), a head-mounted video display device, that is, an information input device using an HMD (Head Mount Display) is known.

  The information input device can input information such as characters to content video such as DVD and video, text video, and the like virtually displayed by the HMD using an input device such as a touch panel or a pointer. Has been made. This makes it possible to input desired information in the virtual reality world by the same operation as in the real world. In recent years, various information input devices have been proposed.

For example, a technology of an information input device that detects the position of a finger for inputting information using a three-dimensional coordinate measuring device and inputs information to a virtually displayed video based on the detected coordinates (see Patent Document 1). ) In addition, even when a touch panel is taken by using a touch panel as an input device, a plurality of light sources are provided at the periphery of the touch panel to detect the position of the touch panel, and each light source is controlled to blink with different blink patterns. There is known an information input device technology (see Patent Document 2) that can easily detect the position of a light source, that is, the position of a touch panel.
Japanese Patent Laid-Open No. 7-5978 JP 2000-172430 A

  However, in the information input device disclosed in Patent Document 1, since the position of a finger for inputting information is detected using a three-dimensional coordinate measuring device, the device is complicated and expensive. In addition, it is considered difficult to detect the finger position with high accuracy. Further, in the information input device disclosed in Patent Document 2, the touch panel as the input device and a plurality of light sources provided at the periphery of the touch panel are controlled to blink with different blink patterns in order to detect the position of the touch panel. Requires a flashing controller. Therefore, there is a risk that the apparatus will be complicated and expensive. In addition, the HMD cannot see through the outside world, and the image that is virtually displayed together with the movement of the wearer's head moves, so that there is a shift in the position between the real world and the virtual world. This makes it impossible to input information at an appropriate position.

  The present invention has been made in view of the above problems, and in an information input device using an HMD, information can be easily and accurately displayed on a virtually displayed image without increasing the complexity and cost of the device. It is an object of the present invention to provide an information input device that can input.

  The above object is achieved by the invention described in any one of 1 to 9 below.

1. Information comprising display means mounted on the head or face and capable of displaying an image and see-through the outside, a virtual display frame in which the image is virtually displayed, and a pointer for inputting information to the virtually displayed image An input device,
Position detecting means for detecting the position of the virtual display frame and the position of the pointer;
Synthesizing means for synthesizing the locus of the position of the pointer detected by the position detecting means with the video;
An information input device comprising: display position control means for controlling a position where the video is virtually displayed according to the position of the virtual display frame detected by the position detection means.

  2. 2. The information input device according to 1 above, wherein the position detection unit includes a camera unit that captures an outside world and has a zoom function.

  3. 3. The information input device according to claim 1 or 2, wherein the position detecting unit detects the position of the virtual display frame and the position of the pointer based on a contrast of an image captured by the camera unit.

4). The virtual display frame is a sheet provided with a frame on its surface,
4. The information input device according to any one of 1 to 3, wherein the light reflectance of the frame and the sheet is different.

  5. 5. The method according to any one of 1 to 4, wherein the synthesizing unit outputs a predetermined video or information to the display unit based on the position of the virtual display frame detected by the position detecting unit. The information input device described.

  6). 2. The information input device according to 1, wherein the pointer is formed in a pen shape.

7). The pointer has a light source at one end,
7. The information input device according to 6, wherein the light source is turned on when the light source comes into pressure contact with an object.

  8). An input mode for combining the locus of the pointer position with the image by the combining means, a full erasing mode for erasing all the locus of the pointer position combined with the image, and a partial erasing mode for erasing a part thereof. 8. The information input device according to any one of 1 to 7, further comprising a mode changeover switch for selecting one of them.

  9. When the partial erasure mode is selected by the mode changeover switch, by tracing the locus of the position of the pointer synthesized with the video and virtually displayed in the virtual display frame by the pointer by the pointer, 9. The information input device according to any one of 1 to 8, wherein the traced part of the locus is deleted.

  According to the present invention, the display position control means controls the position where the video is virtually displayed according to the position of the virtual display frame detected by the position detection means. That is, even when the wearer moves his / her head and the position of the virtual display frame deviates from the position of the virtually displayed image, the display position control unit is configured to detect the virtual display frame detected by the position detection unit. The position where the video is virtually displayed can be moved according to the position of the image. Therefore, the position of the virtual display frame and the position of the virtually displayed video can be matched without being affected by the movement of the wearer's head. Thereby, the position shift between the real world and the virtual world is eliminated, and the wearer can easily and accurately input information to the virtually displayed video.

  In addition, the position detection means includes a camera unit that captures the outside world and has a zoom function. Therefore, the position of the virtual display frame and the position of the pointer can be easily detected based on the video data photographed by the camera unit. In addition, the virtual display frame size and virtual display frame that can be seen through with a zoom function such as optical zoom and electronic zoom, regardless of the distance between the wearer's head position and the virtual display frame, are captured by the camera unit and displayed virtually. The size of the video display frame to be matched can be made coincident.

  Further, the position detection means detects the position of the virtual display frame and the position of the pointer based on the contrast of the video imaged by the camera unit. Therefore, it is possible to easily detect the object with high accuracy.

  Further, since the reflectance of the light of the virtual display frame and the sheet is different, contrast can be provided.

  Further, the synthesizing means outputs predetermined video and information to the display means based on the position of the virtual display frame detected by the position detecting means. Therefore, for example, when the position detection means detects a virtual display frame, the virtual display frame is flashed, or the virtual display frame is photographed with the camera unit to the size of the virtual display frame that can be viewed through by zooming. When the size of the video display frame to be virtually displayed matches, the wearer can easily optimize the shooting angle of view by lighting the video display frame to be virtually displayed. In addition, if the virtual display frame that can be observed through see-through protrudes from the shooting angle of view of the camera unit, a warning is displayed, which indicates that the wearer is improperly adjusting the shooting angle of view and that the position detection accuracy decreases. Can be recognized.

  Moreover, since the pointer is formed in a pen shape, it can be easily held and has excellent operability.

  The light source of the pointer is turned on when the object is pressed against the object. In other words, when the information input operation is not performed and the pointer is not in contact with the object, the light source is not turned on. Therefore, the position detection unit can prevent erroneous input without detecting the position of the pointer. it can.

  In addition, a mode changeover switch for selecting any one of the input mode, the full erase mode, and the partial erase mode is provided. Therefore, the wearer can select a desired operation mode according to the purpose.

  Further, when the partial deletion mode is selected, the traced portion of the locus is erased by tracing the locus of the virtually displayed pointer position with the pointer. Therefore, the wearer can easily delete only desired information according to the purpose.

  Embodiments of an information input device according to the present invention will be described below with reference to the drawings.

  First, the configuration of the information input device 1 will be described with reference to FIG. FIG. 1 is a diagram showing an outline of the overall configuration of an information input device 1 according to the present invention.

  As shown in FIG. 1, the information input device 1 includes an HMD 2, a control unit 3, a virtual display sheet 4, a pointer 5, and the like, and is displayed on a display unit 25 described later provided in the HMD 2 and displayed on the virtual display sheet 4. Information such as characters is input to the virtually displayed video using the pointer 5.

  First, the external appearance of the HMD 2 will be described. The HMD 2 is a head-mounted image display device that is used in the vicinity of the user's eyeball. As shown in FIG. 1, the temples 21R and 21L, the bridge 22, the nose pads 23R and 23L, and the transparent substrate 24R. 24L, display unit 25 (LCD display unit 26, eyepiece optical system 27), camera unit 28, earphones 29R, 29L, and the like. The HMD 2 displays content video such as DVD and video, text video, and the like captured from a memory card 39 (described later) loaded in the control unit 3 on the LCD display unit 26 provided in the display unit 25, and displays the LCD display. By directly projecting the image obtained from the unit 26 onto the eyeball of the wearer by the eyepiece optical system 27, it is possible to observe a virtual image as if the image was magnified and projected in the air. In the present invention, the virtual image is virtually displayed on the virtual display sheet 4.

  The temples 21R and 21L are long members made of a flexible elastic material or the like. The temples 21R and 21L are hooked on the wearer's ears or the temporal region and are held on the head of the wearer by the wearer. This is for adjusting the mounting position. The temples 21R and 21L are configured to be rotatable in the directions of arrows P and Q in the rotating portions 21Ra, a and 21La. When the HMD 2 is not used, the temples 21R and 21L are arranged in the directions of the arrows P and Q. It can be made compact by rotating it along the transparent substrates 24R and 24L.

  The bridge 22 includes nose pads 23R and 23L that hold the HMD 2 on the face of the wearer, and is a short rod-like member that is stretched over a predetermined position on the transparent substrates 24R and 24L facing each other. The transparent substrate 24L is held in a relative positional relationship with a fixed gap.

  The transparent substrates 24R and 24L are substantially flat transparent bodies that form a U-shaped space 24Rs at a position corresponding to one eyeball. An eyepiece optical system 27 is fitted in a U-shaped space 24Rs formed by being surrounded by the transparent substrate 24R.

  The display unit 25 corresponds to the display means in the present invention, and includes an LCD display unit 26, an eyepiece optical system 27, and the like. The display unit 25 is a DVD, a video, or the like captured from a memory card 39 described later loaded in the control unit 3. Display content video, text video, etc.

  The camera unit 28 includes a zoom lens 281 (not shown), a CCD (Charge Coupled Device) 282, an image processing unit 284, and the like. The camera unit 28 captures an external scene around the wearer. The imaged subject optical image is photoelectrically converted by the CCD 282 to generate an image signal (video signal), and the image processing unit 284 or the like performs predetermined image processing on the image signal to generate an image (video).

  Earphones 29R and 29L are inserted into the ears of the wearer and provide sound to the wearer.

  Here, the configuration of the display unit 25 will be described with reference to FIG. FIG. 2 is a side sectional view from the left side surface of the display unit 25 in the information input device 2 according to the present invention, and mainly shows the internal configuration of the display unit 25.

  As shown in FIG. 2, the display unit 25 includes a housing 261, an LED (Light Emitting Diode) 262, a collimator lens 263, an LCD (Liquid Crystal Display) 264 LCD display unit 26, a prism 271, and HOE (Holographic Optical). Element) 272 and an eyepiece optical system 27.

  With the LED 262, the collimator lens 263, and the LCD 264 built in the housing 261 of the LCD display unit 26, the housing 261 is diagonally upward and forward at the upper end of the prism 271 of the eyepiece optical system 27 (FIG. 2). Then, it is attached in such a manner as to protrude obliquely upward to the right.

  The LED 262 is a point light source composed of a white light emitting diode (LED) including a predetermined wavelength color.

  The collimator lens 263 converts the light from the LED 262 into substantially parallel light and projects it onto the LCD 264.

  The LCD 264 generates a video based on a content video signal such as a DVD or video, a text video signal, etc. taken from a memory card 39 (described later) loaded in the control unit 3, and is, for example, a transmissive liquid crystal display panel. is there.

  The prism 271 is a substantially plate-shaped transparent member made of glass, transparent resin, or the like, and causes light from the LCD 264 to be reflected a plurality of times inside. The upper end portion of the prism 271 is formed in a wedge shape so that the front side (opposite side of the eyepiece surface) protrudes to be thicker upward so that most of the light from the LCD 264 can be collected inside. The thick part 271b thus formed is formed.

  An inclined surface 271a is formed at the lower end of the prism 271. The prism 271 is bonded (for example, bonded) to the inclined surface 24Ra formed on the transparent substrate 24R via the HOE 272. The front and back surfaces of the prism 271 are flush with the front and back surfaces of the transparent substrate 24R. Thereby, the prism 271 is integrated with the transparent substrate 24R in a single plate shape.

  The HOE 272 is a volume phase type holographic optical element having a positive power, which is constituted by a so-called free-form surface that is optically asymmetric with respect to the axis, and is supported at a lower end portion of the prism 271 with a predetermined inclination angle. The HOE 272 provides a hologram image to the eyeball E using the light interference phenomenon when irradiated with the light guided by the prism 271.

  In the display unit 25 having the above-described configuration, the light emitted from the LED 262 illuminates the LCD 264 through the collimator lens 263, and the image light generated by the LCD 264 by this illumination is reflected in the prism 271 a plurality of times. After reflection, the light is diffracted by the HOE 272 and guided to the user's eyeball E as a virtual image.

  Further, the prism 271 guides light incident from the front to the user's eyeball E. As a result, the user can see-through the outside world (front subject), and content videos such as DVDs and videos, text images, etc. captured from the memory card 39 described later are superimposed on the outside world (front subject) for visual recognition. Will be.

  The inclined surface 24Ra formed on the transparent substrate 24R cancels (cancels) the light refraction at the inclined surface 271a of the prism 271. That is, the light from the arrow W side upwards due to the prism effect on the inclined surface 271a. In order to prevent bending, the wearer can observe outside light through the prism 271, the transparent substrate 24R, and the HOE 272 without distortion.

  Returning to FIG. 1, the control unit 3 is composed of a microcomputer, and includes a power switch 31, a zoom switch 32, an audio volume 33, an initial setting switch 34, a mode changeover switch 35, etc., and signals from these various operation switches. Accordingly, the display operation of the display unit 25, the photographing operation of the camera unit 28, and the image signal processing operation are comprehensively controlled.

  The initial setting switch 34 performs virtual display for the size of the virtual display frame 41 that can be observed through a see-through prior to performing an input operation of information such as characters on the video virtually displayed on the virtual display sheet 4 using the pointer 5. The frame 41 is used when performing an operation of optimizing the size of the video display frame that is photographed by the camera unit 28 and virtually displayed.

  The mode changeover switch 35 is an input mode for combining the locus of the pointer 5 with the virtually displayed image, an all erasing mode for erasing all of the locus of the pointer 5 from the synthesized image, and erasing a part thereof. One of the partial erase modes to be selected is selected. Details of operations by the initial setting switch 34 and the mode changeover switch 35 will be described later.

  The virtual display sheet 4 is a white sheet in which, for example, a black and rectangular virtual display frame 41 is drawn on the surface, and the video displayed on the display unit 25 is virtually displayed.

  The pointer 5 is a pen-shaped information input device having an LED 51 corresponding to the light source of the present invention at the tip thereof. When the LED 51 is brought into pressure contact with the virtual display sheet 4, the LED 51 is lit. A virtual display frame / pointer coordinate detection unit 371 in the control unit 37 to be described later is based on the coordinates of the virtual display frame 41 and the pointer 5 based on the contrast (luminance difference) of the subject in the video data captured by the camera unit 28. Is detected.

  Next, the electric circuit configuration of the information input device 1 will be described with reference to FIG. FIG. 3 is a block diagram of an electric circuit of the information input device 1 according to the present invention. In FIG. 3, the same members as those shown in FIGS.

  The main electric circuit block of the information input device 1 includes a display unit 25, a camera unit 28, a control unit 3, and the like.

  The display unit 25 includes an LCD display unit 26 and an eyepiece optical system 27. Since the operations performed in each part have been described above, the description thereof will be omitted.

  The camera unit 28 includes a zoom lens 281, a CCD (Charge Coupled Device) 282, an AFE (Analog Front End) 283, an image processing unit 284, and the like.

  The CCD 282 is a color area imaging sensor in which R (red) light, G (green) light, and B (blue) light transmission filters are arranged in a checkered pattern in pixel units (pixel units), and is connected by a zoom lens 281. The imaged subject light image is photoelectrically converted into image signals (signals consisting of a signal sequence of pixel signals received in units of pixels) of each color component of R (red) light, G (green) light, and B (blue) light. To convert.

  The AFE 283 controls the drive of the CCD 282 based on the reference clock transmitted from the control unit 3, performs a well-known analog signal processing on the video signal (image signal) read from the CCD 282, and converts it into a digital video signal To do.

  In this manner, the video signal read by the CCD 282 is subjected to predetermined processing by the AFE 283 and converted into a digital video signal. The digitized video signal is captured by the image processing unit 284 and subjected to predetermined processing. Hereinafter, the process for the digital video signal performed by the image processing unit 284 will be described.

  First, the digital video signal captured by the image processing unit 284 is written into an image memory 381 in the control unit 3 described later in synchronization with the reading of the video signal output from the CCD 282. That is, a digital video signal used for processing performed in the image processing unit 284 is once recorded in the image memory 381 and is taken out from the image memory 381 and used for processing in each part in the image processing unit 284.

  The image processing unit 284 includes, for example, a black level correction unit, a pixel interpolation unit, a white balance control unit, a gamma correction unit, a matrix calculation unit, a shading correction unit, and an image compression unit (not shown). The digital video signal is subjected to known image signal processing. Then, the digital video signal that has undergone predetermined processing at these parts is stored again in the image memory 381.

  In the photographing standby state, the digital video signal is taken into the image memory 381 from the camera unit 28 at a predetermined interval such as every 1/30 seconds. That is, the video data (digital video signal) written in the image memory 381 is updated every 1/30 seconds, for example.

  Next, the control unit 3 includes a control unit 37, an image memory 381, a VRAM (Video Random Access Memory) 382, and the like.

  The control unit 37 reads a ROM (Read Only Memory) that stores each control program (not shown), a RAM (Random Access Memory) that temporarily stores data such as arithmetic processing and control processing, and a control program from the ROM. CPU (central processing unit) etc. that execute the above, and in response to signals from the aforementioned operation switches, control the display operation of the display unit 25, the photographing operation of the camera unit 28, and the image signal processing operation. To do.

  As shown in FIG. 3, the main part of the control unit 37 includes a virtual display frame / pointer coordinate detection unit 371, a video generation unit 372, a video synthesis unit 373, a frame position variation detection unit 374, and a pointer relative position coordinate calculation unit 375. , A buffer memory 376, a display area correction unit 377, and the like.

  The virtual display frame / pointer coordinate detection unit 371 reads the video data captured by the camera unit 28 and written to the image memory 381, and based on the contrast (luminance difference) of the subject in the read video data, the virtual display frame 41. And the coordinates of the pointer 5 are detected.

  The video generation unit 372 shows the virtual display frame 41 based on the coordinates of the virtual display frame 41 detected by the virtual display frame / pointer coordinate detection unit 371 and the locus of the coordinates of the pointer 5 written in the buffer memory 376 described later. The video (video display frame) and the video showing the locus of the pointer 5 are generated.

  The video composition unit 373 includes a video (video display frame) showing the virtual video display frame 41 generated by the video generation unit 372 and a content video such as a DVD and a video captured from the memory card 39 and a text video. Is combined with the video showing.

  The frame position variation calculation unit 374 reads the coordinates of the virtual display frame 41 detected by the virtual display frame / pointer coordinate detection unit 371 and updated every 1/30 seconds, for example, at predetermined intervals, and from the previously read coordinates. Calculate the amount of variation. As described above, the frame position variation calculation unit 374, the camera unit 28, and the virtual display frame / pointer coordinate detection unit 371 function as a position detection unit in the information input device according to the present invention.

  The pointer relative position coordinate calculation unit 375 calculates the relative coordinates of the pointer 5 with respect to the virtual display frame 41 based on the coordinates of the virtual display frame 41 detected by the virtual display frame / pointer coordinate detection unit 371 and the coordinates of the pointer 5.

  The buffer memory 376 stores a locus of the coordinates of the pointer 5 that is calculated by the pointer relative position coordinate calculation unit 375 and changes with time as the pointer 5 moves. As described above, the buffer memory 376, the above-described video generation unit 372, and video synthesis unit 373 function as a synthesis unit in the information input device according to the present invention.

  The display area correction unit 377 is a virtual display calculated by the frame position fluctuation amount calculation unit 374, where the content video such as DVD and video captured from the memory card 39 and the text video are displayed on the LCD display unit 26. The frame 41 is moved in accordance with the amount of change in coordinates. Thus, the display area correction unit 377 and the frame position variation calculation unit 374 described above function as display position control means in the information input device according to the present invention.

  Next, the image memory 381 is a temporary memory used as a work area for performing various processing on the digital video signal by the image processing unit 284 in the camera unit 28 and the control unit 37 in the control unit 3.

  The VRAM 382 has a video signal recording capacity corresponding to the number of pixels of the LCD 264 in the LCD display unit 26, and is a video signal buffer memory that constitutes a video to be reproduced and displayed on the LCD 264.

  In the information input device 1 having such a configuration, the present invention can easily and accurately input information to a content video such as a DVD or a video, a text video, or the like virtually displayed in the virtual display frame 41. It is to make. Specifically, even when the wearer moves his / her head and the position of the virtual display frame 41 is shifted from the position of the virtually displayed image, the virtual display frame 41 detected by the position detection unit is displayed. The video display operation performed by the information input device 1 is controlled so that the position where the video is virtually displayed is moved in accordance with the position so that the position of the video displayed virtually matches the position of the virtual display frame 41. is there. As a result, the positional shift between the real world and the virtual world is eliminated, and information can be easily and accurately input to the virtually displayed video.

  Here, details of the video display operation performed in the information input apparatus 1 will be described. The present invention is suitable for inputting information such as text into a document that needs to be considered confidentiality such as tests and questionnaires. Here, the flow of the video display operation control performed by the information input device 1 when the test question is virtually displayed and the answer is written in the virtually displayed video will be described.

  First, the initial setting operation will be described with reference to FIG. 4 and FIG. FIG. 4A is a schematic diagram showing the relationship between the shooting angle of view of the camera unit 28 and each display area. 4B1 to 4D2 are schematic diagrams illustrating display areas based on the shooting angle of view of the camera unit 28. FIG. FIG. 8 is a flowchart showing the flow of the initial setting operation.

  The initial setting operation is a virtual display frame with respect to the size of the virtual display frame 41 that can be observed through the see-through prior to performing an input operation of information such as characters on the video virtually displayed on the virtual display sheet 4 using the pointer 5. 41 is an operation performed for optimizing the size of the video display frame that is captured by the camera unit 28 and virtually displayed.

  First, a virtual display sheet 4 on which a virtual display frame 41 on which a video showing a test question stored in the memory card 39 is virtually displayed is drawn on a desk.

  In FIG. 4A, an LCD display area L indicates a maximum display area (maximum display range) of an image virtually displayed on a desk. The shooting angle of view K indicates the shooting angle of view (shooting range) of the camera unit 28. A virtual display frame W indicates a virtual display frame 41 that can be observed through see-through. The video display frame D indicates an area (range) where video is virtually displayed.

  Here, the flow of the initial setting operation will be described with reference to the flowchart shown in FIG.

  First, the wearer views the virtual display frame W (virtual display frame 41) drawn on the virtual display sheet 4 on the desk in a posture of sitting on a chair and taking a normal test, and presses the initial setting switch 34 (step S1). . When the initial setting switch 34 is turned on, the camera unit 28 starts the live view shooting operation and takes the video into the image memory 381 (step S2). When the video is captured in the image memory 381, the virtual display frame / pointer coordinate detection unit 371 reads the video data written in the image memory 381, and based on the contrast (luminance difference) of the subject in the read video data, The coordinates of the display frame W are detected (step S3). When the coordinates of the virtual display frame W are detected, the video generation unit 372 generates a video (video display frame D) indicating the virtual display frame W based on the detected coordinates of the virtual display frame W, and blinks on the LCD 264. It is displayed (step S4). Next, the video generation unit 372 has generated in advance, “Please operate the zoom switch and press the initial setting switch after aligning the flashing video display frame with the virtual display frame that can be viewed through”. Is displayed on the LCD 264 (step S5). The wearer compares the virtual display frame W that can be observed through see-through with the video display frame D that is blinking, and operates the zoom switch 32 to match the positions (sizes) of the two. The initial setting switch 34 is pushed again. When the initial setting switch 34 is pressed again (step S6; Yes), the frame position variation calculation unit 374 stores the coordinates of the four corners of the virtual display frame W detected by the virtual display frame / pointer coordinate detection unit 371. (Step S7). Then, the video generation unit 372 lights up and displays the blinking video display frame D (step S8). When the video display frame D changes to lighting display, the video composition unit 373 reads the video indicating the test question stored in the memory card 39 and displays it on the LCD 264 in accordance with the video display frame D. On the other hand, when there is no input from the initial setting switch 34 for a predetermined time (step S10; Yes), the video generation unit 372 turns off the blinking video display frame D (step S11) and performs error display or initial setting again. A prompt message is displayed on the LCD 264, and the initial setting operation is terminated.

  Here, the relationship between the shooting field angle K of the camera unit 28 by the operation of the zoom switch 32 and the video displayed in the video display frame D will be described with reference to FIGS. 4B1 to 4D2.

  As shown in FIG. 4 (b1), when the shooting angle of view K of the camera unit 28 is larger than the LCD display area L, the video display frame D becomes relatively small, and as shown in FIG. The video of the test question displayed in the display frame D becomes smaller. On the other hand, as shown in FIG. 4 (d1), when the shooting field angle K of the camera unit 28 is smaller than the virtual display frame W, the virtual display frame W cannot be detected, and as shown in FIG. 4 (d2). The video cannot be displayed. As shown in FIG. 4 (c1), when the shooting angle of view K of the camera unit 28 and the LCD display area L match, that is, when the virtual display frame W and the video display frame D match, FIG. As shown, the size of the test question image displayed in the image display frame D is optimized.

  Further, the state of the image displayed when the distance between the wearer's head and the desk (virtual display frame 41) changes after the photographing field angle K of the camera unit 28 is optimized by the initial setting operation is illustrated. 5 will be described. 5A shows an image when the wearer's head approaches the desk (virtual display frame W), and FIG. 5B shows the distance between the wearer's head and the desk (virtual display frame W). FIG. 5C is a schematic diagram showing an image when the wearer's head is moved away from the desk (virtual display frame W).

  As shown in FIGS. 5A to 5C, after the initial setting is properly performed, when the distance between the wearer's head and the desk (virtual display frame W) changes, the LCD display area L (camera unit 28) is changed. The size of the field angle of view K) does not change, and when the size of the virtual display frame W on the desk approaches, it increases, and when it moves away, it changes small. Since the camera unit 28 is fixed to the wearer's head, as shown in FIG. 5A, when the head approaches the desk (virtual display frame W), the camera unit 28 has a shooting angle of view K. When the virtual display frame W on the desk becomes large and the virtual display frame W exceeds the shooting angle of view K (shooting range) of the camera unit 28, the virtual display frame W cannot be detected. On the other hand, as shown in FIG. 5C, when the head is moved away from the desk (virtual display frame W), the virtual display frame W on the desk becomes smaller with respect to the shooting angle of view K of the camera unit 28. Since the position of the virtually displayed video is controlled based on the coordinates of the virtual display frame W detected by the camera unit 28, the size changes according to the virtual display frame W on the desk. Therefore, since the test question looks as if it is displayed on the desk, a sense of reality can be obtained.

  Next, the information input operation will be described with reference to FIG. 6, FIG. 7, and FIG. FIGS. 6A to 6D are schematic diagrams showing how video changes due to an information input operation. 7A to 7C are schematic diagrams showing the relationship between the head movement of the wearer, the virtual display frame W, and the video display frame D. FIG. FIG. 9 is a flowchart showing the flow of information input operation.

  In the present invention, as shown in FIG. 7A, the position of the video display frame D with respect to the position of the virtual display frame W that has been optimized is changed, for example, by moving the head of the wearer. As shown in FIG. 7, even in the case of deviation, as shown in FIG. 7C, the position of the video display frame D is moved in accordance with the position of the virtual display frame W detected by the position detection unit, and the virtual display is performed. The video display operation performed by the information input device 1 is controlled so that the position of the video display frame D matches the position of the frame W.

  Here, the flow of the information input operation will be described with reference to the flowchart shown in FIG.

  First, the mode changeover switch 35 is operated to set the input mode (step S1). When the mode changeover switch 35 is set to the input mode, the camera unit 28 starts the live view shooting operation and takes the video into the image memory 381 (step S2). When the video is captured in the image memory 381, the virtual display frame / pointer coordinate detection unit 371 reads the video data written in the image memory 381, and based on the contrast (luminance difference) of the subject in the read video data, The coordinates (P, Q, R, S) of the four corners of the display frame W are detected (step S3). When the coordinates of the virtual display frame W are detected, the video generation unit 372 displays the video indicating the virtual display frame W based on the coordinates of the four corners of the detected virtual display frame W, as shown in FIG. (Video display frame D) is generated and displayed on the LCD 264 (step S4). Then, as shown in FIG. 6B, the video composition unit 373 reads the video indicating the test problem stored in the memory card 39 and displays it on the LCD 264 in accordance with the video display frame D (step S5).

  Next, the virtual display frame / pointer coordinate detection unit 371 reads the video data written in the image memory 381, and based on the contrast (luminance difference) of the subject in the read video data, as shown in FIG. Then, the coordinate (T) of the pen tip of the pointer 5 is detected (step S6). As described above, since the video data (digital video signal) written in the image memory 381 is updated, for example, every 1/30 seconds, the virtual display detected by the virtual display frame / pointer coordinate detection unit 371. The coordinates (P, Q, R, S) of the four corners of the frame W and the coordinates (T) of the pen tip of the pointer 5 are also updated every 1/30 seconds.

  Next, the frame position variation calculation unit 374 compares the four corner coordinates (P, Q, R, S) of the detected virtual display frame W with the previously detected coordinates, and as a result of comparison, the coordinates vary. If not (step S7; NO), the display area correction unit 377 does not move the position of the video display frame D (step S10). Then, the video composition unit 373 displays the video indicating the test problem stored in the memory card 39 on the LCD 264 in accordance with the read video display frame D (step S11). When the video is displayed on the LCD 264 in accordance with the video display frame D, the video generation unit 372 is calculated by the pointer relative position coordinate calculation unit 375 (step S12), and the pointer 5 changes with time as the pointer 5 moves. Is read from the buffer memory 376, and an image PT showing the locus of the pointer 5 is generated. Then, as shown in FIG. 6D, the video composition unit 373 synthesizes the generated video PT indicating the locus of the pointer 5 with the video indicating the test problem displayed in the video display frame D and displays it on the LCD 264. (Step S13).

On the other hand, in step S7, the frame position variation calculation unit 374 compares the four corner coordinates P, Q, R, and S of the detected virtual display frame W with the previously detected coordinates, and as a result of comparison, the coordinates are When it fluctuates (step S7; Yes), the fluctuation amount is calculated (step S8). Specifically, the fluctuation amounts of the coordinates P, Q, R, and S at the four corners of the virtual display frame W are ΔP, ΔQ, ΔR, and ΔS, and the head is on the right side of the virtual display frame W, for example, the X coordinate In the case of 10 movements, the variation amount of the X coordinate and the variation amount of the Y coordinate of ΔP, ΔQ, ΔR, and ΔS are the values of the following (Equation 1) and (Equation 2), respectively.
ΔPx = ΔQx = ΔRx = ΔSx = −10 (Formula 1)
ΔPy = ΔQy = ΔRy = ΔSy = 0 (Formula 2)
When the fluctuation amount is calculated, the display area correction unit 377 moves the position of the video display frame D according to the calculated fluctuation amount (step S9). Here, the position of the video display frame D is moved to the left (−10 in the X coordinate). Then, the video composition unit 373 displays the video indicating the test question stored in the memory card 39 on the LCD 264 in accordance with the read video display frame D (step S11).

Further, when the head moves in a direction away from the virtual display frame W, for example, when moving away from the surface of the virtual display frame W, the virtual display frame W recognized by the camera unit 28 becomes small and In contrast, it moves in the inner direction. At this time, if the variation amount of the X coordinate and the Y coordinate is 5, respectively, the variation amount of the X coordinate of ΔP, ΔQ, ΔR, and ΔS and the variation amount of the Y coordinate are values of the following (Equation 3) to (Equation 6), respectively. It becomes.
ΔPx = ΔRx = 5 (Formula 3)
ΔQx = ΔSx = −5 (Formula 4)
ΔPy = ΔQy = −5 (Formula 5)
ΔRy = ΔSy = 5 (Formula 6)
Since the video of the test question is always displayed with the four corners of coordinates P, Q, R, and S, in this case, it is reduced compared to the previous time.

  Next, the pointer relative position coordinate calculation unit 375 calculates the display position of the pointer 5 (step S12). As described above, the video display frame D is a parallel movement, enlargement / reduction, a combination thereof, or a complicated change. However, since the movement is linked with the camera unit 28, a large error occurs. Does not occur. However, the pointer 5 is linked to the movement of the wearer's hand and is independent of the movement of the head. Therefore, when the position of the pointer 5 is recognized by the absolute value of the coordinate T, correction is difficult and accuracy is also deteriorated. That is, this problem can be solved by obtaining relative positions with respect to the coordinates P, Q, R, and S of the four corners of the virtual display frame W instead of absolute values.

  Specifically, it is obtained as follows from the coordinates P, Q, R, and S of the four corners of the virtual display frame W recognized by the camera unit 28.

First, the coordinates T (Tx, Ty) of the pointer 5 with respect to the coordinates P (Px, Py), Q (Qx, Qy), R (Rx, Ry), and S (Sx, Sy) of the four corners of the virtual display frame W The ratio α in the left-right direction (X-axis direction) and the ratio β in the up-down direction (Y-axis direction) are calculated from the following (Expression 7) and (Expression 8).
α: (Tx−Px) / (Qx−Px) (Formula 7)
β: (Ty-Ry) / (Py-Ry) (Formula 8)
From the ratio α in the horizontal direction (X-axis direction) of the coordinates T (Tx, Ty) of the pointer 5 calculated from (Expression 7) and (Expression 8), and the ratio β in the vertical direction (Y-axis direction), the pointer 5 The relative position coordinates T ′ (Tx ′, Ty ′) with respect to the coordinates P, Q, R, and S of the four corners of the virtual display frame W are obtained by the following (Expression 9) and (Expression 10).
Tx ′ = α {(Qx + ΔQx) − (Px + ΔPx)} + Px + ΔPx (Equation 9)
Ty ′ = β {(Py + ΔPy) − (Ry + ΔRy)} + Ry + ΔRy (Formula 10)
The pointer relative position coordinate calculation unit 375 records the relative position coordinate T ′ obtained in this way in the buffer memory 376. The video generation unit 372 reads the relative position coordinate T ′ of the pointer 5 from the buffer memory 376 and generates a video PT indicating the locus of the pointer 5 at the position of the relative position coordinate T ′. Then, the video composition unit 373 synthesizes the generated video PT indicating the locus of the pointer 5 with the video indicating the test problem displayed in the video display frame D and causes the LCD 264 to display and display the video (step S13). As described above, for example, since the camera unit captures an image every 1/30 seconds, the coordinate T ′ of the pointer 5 is displayed as a locus by repeating the flow described with reference to FIG.

  As described above, in the present invention, information can be easily and accurately input to content video such as DVD and video, text video, and the like virtually displayed in the virtual display frame W. That is, even when the position of the virtual display frame W and the position of the video display frame D are shifted due to the wearer moving his / her head, the wearer moves the head according to the position of the virtual display frame W detected by the position detection unit. The position of the video display frame D can be matched with the position of the virtual display frame 41 by moving the position of the video display frame D. As a result, the positional shift between the real world and the virtual world is eliminated, and information can be easily and accurately input to the virtually displayed video.

  In the information input device 1 according to the present invention, it is possible to delete all information such as characters input to content video such as DVD and video, text video, or a part of the information. . Specifically, when the mode changeover switch 35 is operated to set the all erase mode, the locus of the coordinates of the pointer 5 stored in the buffer memory 376 is all erased, and information such as characters combined with the video is displayed. Everything is erased. Further, the mode changeover switch 35 is operated to set the partial deletion mode, and the video PT showing the locus of the position of the virtually displayed pointer 5 is traced by the pointer 5 to correspond to the traced portion of the video. The locus of the coordinates of the pointer 5 stored in the buffer memory 376 is deleted, and a part of the information such as characters synthesized in the video is deleted. Thereby, the wearer can easily erase only desired information according to the purpose.

  The present invention has been described above with reference to the embodiments. However, the present invention should not be construed as being limited to the above-described embodiments, and can be changed or improved as appropriate. For example, in the above-described embodiment, the shooting field angle K of the camera unit 28 is adjusted using the zoom lens 281, but a wide-angle lens is used instead of the zoom lens 281, and wide-angle shooting is performed using the wide-angle lens. The angle of view may be adjusted by, for example, enlarging the image by electronic zoom (digital zoom).

1 is an overall configuration diagram of an information input device according to the present invention. It is a sectional side view of the display unit in the information input device concerning the present invention. It is an electric circuit block block diagram of the information input device which concerns on this invention. It is a schematic diagram which shows the relationship between the imaging | photography field angle of a camera unit and each display area in the information input device which concerns on this invention. It is a schematic diagram which shows the relationship between the distance from a wearer's head to a virtual display sheet, and the image displayed virtually. It is a schematic diagram which shows the mode of the change of the image | video accompanying information input operation | movement. It is a schematic diagram which shows the relationship between a wearer's head movement, a virtual display frame, and a video display frame. It is a flowchart which shows the flow of initial setting operation | movement. It is a flowchart which shows the flow of information input operation | movement.

Explanation of symbols

1 Information input device 2 HMD
21R, 21L Temple 22 Bridge 23R, 23L Nose pad 24R, 24L Transparent substrate 25 Display unit 26 LCD display unit 261 Case 262 LED
263 Collimator lens 264 LCD
27 Eyepiece optical system 271 Prism 272 HOE
28 Camera Unit 281 Zoom Lens 282 CCD Area Sensor 283 AFE
284 Image processing unit 287 Zoom motor 29R, 29L Earphone 3 Control unit 31 Power switch 32 Zoom switch 33 Audio volume 34 Initial setting switch 35 Mode switch 36 Cable 37 Control unit 371 Virtual display frame / pointer coordinate detection unit 372 Video generation unit 373 Video composition unit 374 Frame position variation calculation unit 375 Pointer relative position coordinate calculation unit 376 Buffer memory 377 Display area correction unit 378 Zoom motor drive circuit 379 Audio output circuit 381 Image memory 382 VRAM
39 Memory Card 4 Virtual Display Sheet 41 Virtual Display Frame 5 Pointer 51 LED
E eyes

Claims (9)

Information comprising display means mounted on the head or face and capable of displaying an image and see-through the outside world, a virtual display frame in which the image is virtually displayed, and a pointer for inputting information to the virtually displayed image An input device,
Position detecting means for detecting the position of the virtual display frame and the position of the pointer;
Synthesizing means for synthesizing the locus of the position of the pointer detected by the position detecting means with the video;
An information input device comprising: display position control means for controlling a position where the video is virtually displayed according to the position of the virtual display frame detected by the position detection means. The information input device according to claim 1, wherein the position detection unit includes a camera unit that captures an external environment and has a zoom function. 3. The information input device according to claim 1, wherein the position detection unit detects the position of the virtual display frame and the position of the pointer based on a contrast of an image photographed by the camera unit. The virtual display frame is a sheet provided with a frame on its surface,
The information input device according to claim 1, wherein the frame and the sheet have different light reflectivities. 5. The display device according to claim 1, wherein the synthesizing unit outputs a predetermined image or information to the display unit based on the position of the virtual display frame detected by the position detection unit. The information input device described in 1. The information input device according to claim 1, wherein the pointer is formed in a pen shape. The pointer has a light source at one end,
The information input device according to claim 6, wherein the light source is turned on when the light source is brought into pressure contact with an object. An input mode for combining the locus of the pointer position with the image by the combining means, a full erasing mode for erasing all the locus of the pointer position combined with the image, and a partial erasing mode for erasing a part thereof. The information input device according to claim 1, further comprising a mode changeover switch for selecting one of them. When the partial erasure mode is selected by the mode changeover switch, by tracing the locus of the position of the pointer synthesized with the video and virtually displayed on the virtual display frame by the pointer by the pointer, 9. The information input device according to claim 1, wherein the traced portion of the locus is deleted.

Images