JP2000172430A - Information input device, device and method for position recognition thereof, virtual image stereoscopic composition device, and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily recognize the position of an information input tool such as a touch panel by an image processing system and to put a virtual image of an image display device, etc., together with a virtual image of the information input tool. SOLUTION: The touch panel 1 with a reference surface setting function capable of picking up an image is equipped with four light emitting diodes LED1 to LED4 which are fitted at specific positions in the peripheral area of its information input surface and a blink control circuit 13 which controls the input and output of the light emitting diodes LED1 to LED4, and this blink control circuit 13 performs blink control so that the blink patterns of the light emitting diodes LED1 to LED4 are different. When an image of the touch panel 1 is picked up by a special photography device like a panning CCD device, this constitution makes it easy to specify the positions of the light emitting diodes LED1 to LED4 as compared with a case when the illumination patterns of the light emitting diodes LED1 to LED4 are brought under non-blinking control.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an information input tool position recognition mechanism that allows an image processing system or the like to recognize the location of a touch panel or the like, and an information input device suitable for use in a virtual image processing apparatus or the like using the same. The present invention relates to a position recognizing device, a position recognizing method, a virtual image three-dimensional composition device, and a storage medium.

More specifically, a plurality of light sources having different blinking patterns are attached to specific positions of an information input tool such as a touch panel capable of imaging, and the position of the information input tool can be easily recognized by an image processing system based on the light sources. In addition, a virtual image of an image display device or the like can be combined with the virtual image of the information input tool.

[0003]

2. Description of the Related Art In recent years, with the improvement of display technology based on virtual reality, a virtual image three-dimensional synthesizing apparatus for providing an operator with virtual reality over a plurality of image display surfaces has appeared. are doing.

[0004] This type of stereoscopic display device is disclosed in Japanese Unexamined Patent Publication No. 9-23.
7353 can be found in the technical literature. According to this technical document, a projection space of about several meters in length and width is provided, and a display device is arranged on each surface, and a dinosaur,
Images of virtual bodies such as monsters and weapons are displayed in three dimensions. Then, when the operator puts on glasses with a liquid crystal shutter and stands in the projection space, it is as if he were at the same place as the virtual body displayed on each display device.

[0005] In addition, a weapon picked up by an operator in a virtual space is imaged by a camera, and image processing is performed so that a virtual body responds to the movement of the weapon. As a result, the operator can slip back in time to the primitive age several thousand years ago and perform dinosaur extermination, etc., as if playing a game.

[0006]

According to the conventional method of recognizing an information input device, an image processing system attempts to recognize an information input device such as a weapon that an operator holds in real space. In such a case, a background image including the information input device is captured, a contour portion of the information input device is extracted, and a reference pattern for the information input device stored in advance and a contour pattern of the information input device are compared. I have.

Therefore, when only the information input device is recognized by the image processing system and an attempt is made to synthesize a virtual body image or the like with the information input device in the virtual space, the amount of calculation for pattern recognition of the image processing system is required. And the load on the arithmetic unit increases.

Thus, for example, a liquid crystal display or a CRT device as an image display device virtually exists on an information input tool such as a touch panel in the real space to which the operator belongs, and the information input tool in the real space is provided. If it is attempted to configure a virtual image stereoscopic synthesis device that controls display of an image display device in a virtual space based on the above operation, if the conventional pattern recognition method is applied as it is, an information input device in an image processing system Because it is difficult to easily recognize the virtual image, the pattern recognition processing becomes large, the image processing for recognizing the reference plane becomes complicated, and the calculation amount at that time increases, There is a problem that this leads to an increase in the cost of a three-dimensional synthesis device and the like.

Accordingly, the present invention has been made in view of the above-mentioned problems, and enables an image processing system to easily recognize the position of an information input tool such as a touch panel, which can be imaged. An object of the present invention is to provide an information input device capable of synthesizing a virtual image such as an image display device with a virtual image, a position recognition device thereof, a position recognition method thereof, a virtual image three-dimensional synthesis device, and a storage medium.

[0010]

An object of the present invention is to provide an image capturing information input device having an information input surface, comprising at least a plurality of light sources mounted at specific positions in a peripheral area of the information input surface. Control means for controlling the input / output of the light source, and the control means performs the blinking control so that the blinking pattern of the light source is different.

According to the information input device of the present invention, the plurality of light sources attached to specific positions in the peripheral area of the information input surface include:
Since the blinking control is performed by the control means so as to make the blinking pattern different, when the information input device is imaged by a special photographing device such as a panning CCD device, the lighting pattern of a plurality of light sources is non-blinking controlled. In comparison, the position of the light source can be easily specified.

Therefore, since the position of the information input device can be easily recognized by the image processing system from the position of the light source, it is possible to combine an image of an arbitrary image display device with the information input device in a virtual space. It can be sufficiently applied to a virtual image three-dimensional synthesis device and the like.

A position recognition device for an information input device according to the present invention is a device for recognizing the position of an information input device in which a plurality of light sources having different blinking patterns are attached to a specific position. And an arithmetic unit for performing image processing of a luminance signal of a light source by the imaging unit to obtain a position of each of the light sources.

According to the position recognition device of the information input device according to the present invention, with respect to the blinking pattern of the light source of the information input device which is imaged so as to flow in a predetermined imaging direction, the luminance information relating to the blinking pattern is calculated by the arithmetic means. Since the position of the light source is obtained by the image processing, the position of the light source attached to the information input device can be easily specified.

Therefore, since the position of the information input device can be easily recognized by the image processing system from the position of the light source, an arbitrary input tool function such as a keyboard can be provided for the information input device in the virtual space. The present invention can be sufficiently applied to an arbitrary CRT device as a monitor for an information input device, a virtual image three-dimensional synthesizing device for synthesizing a virtual image such as a liquid crystal display, and the like.

In the position recognition method for an information input device according to the present invention, a plurality of light sources having different blinking patterns are attached to a specific surface of an information input device capable of capturing an image, and the light sources having different blinking patterns are provided in a predetermined imaging direction. An image is taken so as to flow, and brightness information of the imaged light source is processed to obtain positional information of each of the light sources.

According to the position recognition method of the information input device of the present invention, light sources having different blinking patterns are imaged so as to flow in a predetermined photographing direction. The position of the light source can be easily specified. Therefore, since the position of the information input device can be easily recognized by the image processing system from the position of the light source, an image in which a virtual image such as a character or an image display device is synthesized by the information input device in the virtual space. Processing can be performed with good reproducibility.

A virtual image three-dimensional synthesizing device according to the present invention is a device for stereoscopically synthesizing an image of a virtual body with an external image to which an operator belongs. Position recognition means for recognizing the information input means in the real space to which the operator belongs, and synthesis means for synthesizing an arbitrary virtual image with an image in the virtual space of the information input means recognized by the position recognition means. , The position recognition means is attached to a specific position of the information input means, at least,
Three or more light sources that blink so that the blinking patterns are different, an imaging means for panning to image the light sources so as to flow in a predetermined imaging direction, and image processing of a luminance signal of the blinking pattern by the imaging means. And calculating means for obtaining the reference plane by connecting the positions of the three light sources, and thereafter connecting the positions of the three light sources.

According to the virtual image three-dimensional synthesizing device of the present invention, the above-described information input device, its position recognizing device and its position recognizing method are applied, and the light source attached to a specific position in the peripheral area of the information input surface is applied. Since the position of the information input device can be easily recognized by the image processing system from the position, the input tool function such as a keyboard having an arbitrary key arrangement can be given to the information input means in the virtual space, or the information input can be performed. An arbitrary CRT device as a monitor for the means, a virtual image processing device capable of synthesizing a virtual image such as a liquid crystal display or the like to perform image processing can be configured.

A recording medium according to the present invention is a recording medium storing an algorithm for stereoscopically combining an image of a virtual body with an external image to which an operator belongs, and at least the recording medium has at least the recording medium of the operator. An algorithm for recognizing information input means capable of being imaged in the real space to which the information input means belongs and storing a virtual image of an arbitrary image display device with an image in the virtual space of the recognized information input means is stored. It is assumed that.

According to the storage medium of the present invention, for example,
When controlling the virtual image three-dimensional synthesis device, the algorithm is read, information input means capable of being imaged in the real space to which the operator belongs is recognized, and any information input means in the recognized virtual space is recognized. Since a virtual image such as an image display device is synthesized, a virtual image three-dimensional synthesis device that synthesizes a character or a virtual image such as an image display device on information input means in a virtual space can be configured with high reproducibility. .

[0022]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an information input device, a position recognizing device, a position recognizing method, a virtual image three-dimensional synthesizing device, and a storage medium according to embodiments of the present invention will be described with reference to the drawings.

(1) Information Input Device as an Embodiment FIG. 1 is a perspective view showing a configuration example of an information input device as an embodiment according to the present invention. In this embodiment, a plurality of light sources having different blinking patterns are attached to specific positions of an information input tool such as a touch panel capable of imaging, and the position of the information input tool can be easily recognized by an image processing system based on the light sources. In addition, a virtual image of an image display device or the like can be combined with the virtual image of the information input tool.

An information input device according to the present invention is an image-capturable input tool having an information input surface, such as a virtual image three-dimensional synthesizing device for stereoscopically synthesizing a virtual body image with an external image to which an operator belongs. It is easy to recognize in an image processing system. Here, the image capturing information input device 1 is a touch panel or the like having a shape that partitions a space, and is a device that can be panned with a special imaging device such as a panning CCD device. This panning shot refers to a shooting mode in which a signal charge is read out from a photoelectric conversion element (such as a photodiode) a plurality of times during the same field period in a panning CCD device.

Touch panel 1 as this information input device
Has a horizontal length of about 30 cm and a vertical length of about 20 to 25 cm. This touch panel 1 is shown in FIG.
1A. A touch plate portion 11 constituting an information input surface is provided on an upper surface of the housing 1A, and when the information input surface is touched, contact position information is output. For example, a matrix-like touch switch array 6 (not shown) is attached to the touch plate portion 11 so that when a switch at a touched position is turned on, the contact position information thereof can be read.

A jog dial 12 is provided on the side of the housing 1A so that operation information other than the contact position information can be input. The transmitter 1 is provided on the upper surface of the housing 1A
4 is provided so that contact position information by the touch plate unit 11 and operation information by the jog dial 12 can be modulated and transmitted to a predetermined transmission signal. As the transmission unit 14, a wireless transmitter, an infrared light emitter, or the like is used.

The touch plate portion 11 forms a reference surface, and is preferably formed in a flat shape without irregularities. At the four corners of the surface of the touch plate section 11, light-emitting diodes LED1 to LED4 are respectively attached as a plurality of light sources, and four points P1 to P4 for setting a reference plane are provided.
(X1, y1), (x2, y2), (x
(3, y3) and (x4, y4) (corresponding to a reference plane in the virtual space where a virtual image such as the liquid crystal display 5 or a CRT device is to be synthesized).

The four light emitting diodes LED1 to LE
D4 has a function as a mark part.
That is, blinking control is performed so that at least the blinking pattern is different so that the mounting position is clear. The blinking patterns of the light emitting diodes LED1 to LED4 are imaged so as to flow in a predetermined imaging direction by a panning CCD device attached to a virtual image stereoscopic synthesis device called a special glasstron. This panning is for specifying the above-mentioned reference plane from the mounting positions of the four light emitting diodes LED1 to LED4. The specification of the reference plane will be described with reference to FIGS.

A blinking control circuit 13 is provided in the housing 1A as control means, and the light emitting diodes LED1 to LE
The input / output of D4 is controlled. For example, in the blinking control circuit 13 shown in FIG. 2, in order to make the image processing system recognize the position of the touch panel 1 with good reproducibility, the light emitting diode LE is used.
Blinking control is performed so that the blinking patterns of D1 to LED4 are different.

In this example, the flicker control circuit 13 is formed as an IC chip, and this IC chip is incorporated in the housing 1A.
A predetermined voltage is applied to each of the light emitting diodes LED1 to LED4 to perform blink control. This blinking control circuit 13 has, for example, a clock generator 61. The clock generator 61 includes a 1/2 frequency divider 62, a 1/3 frequency divider 63, 1
A / 4 frequency dividing circuit 64 is connected, and a clock signal CLK1 of a predetermined frequency and this clock signal CLK1 are divided by 1 /.
Clock signal CLK2 that has been frequency-divided by で in frequency divider circuit 62
And a clock signal CLK3 frequency-divided by 1/3 in the 1/3 frequency dividing circuit 63 and a clock signal CLK4 frequency-divided by the 1/4 frequency dividing circuit 64.

Each of the clock signals CLK1 to CLK4 is supplied to each light emitting diode LED through a stabilizing resistor R.
1, LED2, LED3 and LED4. In this example, power is supplied to the clock generator 61 from the drive power supply of the touch panel 1.

FIG. 3 shows a light emitting diode LED1, LED
FIG. 2 is a waveform diagram showing an example of a blinking pattern of LED3 and LED4. In this example, the non-divided clock signal CLK1 is supplied to the light emitting diode LED1, the clock signal CLK2 obtained by dividing the clock signal CLK1 by １ ／ is supplied to the light emitting diode LED2, and the light emitting diode LE
A clock signal CLK3 obtained by dividing the clock signal CLK1 by 1/3 is supplied to D3, and a clock signal CLK4 obtained by dividing the clock signal CLK1 by 1/4 is supplied to the light emitting diode LED4. Therefore, the blinking patterns of the four light emitting diodes LED1, LED2, LED3, and LED4 can be controlled differently.

As described above, according to the touch panel 1 of the present embodiment, the four light emitting diodes LED1 to LED4 attached to the four corners of the surface of the touch plate 11
Is controlled by the blinking control circuit 13 so as to make the blinking pattern different, so that when the touch panel 1 is imaged by a special photographing device such as a panning CCD device, the lighting pattern of the plurality of light sources is not blinked. The positions of the four light emitting diodes LED1 to LED4 can be easily specified as compared with the case where the above operation is performed.

Therefore, the four light emitting diodes LED
Since the position of the touch panel 1 can be easily recognized by the image processing system from the positions of the LEDs 1 to 4, the image of a liquid crystal display 5 or an arbitrary image display device such as a CRT device is synthesized on the touch panel 1 in a virtual space. The present invention can be sufficiently applied to such a virtual image three-dimensional synthesis apparatus.

(2) A virtual image three-dimensional synthesizing apparatus as a first embodiment to which a position recognition device of an information input device is applied FIG. 4 shows a configuration of a virtual image three-dimensional synthesizing apparatus 100 as a first embodiment according to the present invention. It is a perspective view showing an example. In this embodiment, a plurality of light sources having different blinking patterns are attached to a specific surface of an information input device capable of imaging, and an image is taken so that the light source of the information input device flows in a predetermined imaging direction. Is processed by image processing to obtain the position of the light source, and the virtual image of the image display device is synthesized on the reference plane of the information input device connecting the position of the light source in the virtual space.

The virtual image three-dimensional synthesizing apparatus 100 shown in FIG.
This is a device that stereoscopically combines and displays image images of an arbitrary image display device such as a T device. The virtual image three-dimensional composition device 100 includes a touch panel 1 with a reference plane setting function, a special glasstron 2, and an image processing device 3.

Touch panel 1 as this information input device
Is used, for example, in front of an operator. In this example, the touch panel 1 may be placed on a desk or the like in front of the operator such that the touch plate unit 11 falls within the imaging range of the special glasstron 2. The body 2 of this special glasstron 2
1, a belt 22 is provided, and the main body 21 is worn on the face of the operator like wearing glasses, and the belt 22 is fixed along the outer periphery of the operator's head.

The special glasstron 2 is provided with at least a panning CCD device 23 and a display means 24. Normal CC depending on the model of Special Glasstron 2
A D imaging device 25 is provided. The touch panel 1, the panning CCD device 23, and the image processing device 3 constitute a position recognition mechanism 4 as a position recognition device, and can recognize a reference surface of the touch panel 1 in a real space to which an operator belongs. .

In this example, when a two-dimensional imaging device of an interline transfer system having a vertical transfer unit is used as the panning CCD device 23, signals are sent from the photoelectric conversion element to the vertical transfer unit a plurality of times during the same field period. The charge is read. When a two-dimensional frame transfer type imaging device having a charge storage unit is used as the panning CCD device 23, signal charges are read from the photoelectric conversion element to the charge storage unit a plurality of times during the same field period.

Further, an image processing device 3 is connected to the special glasstron 2, and image processing for recognizing a reference plane or the like is performed based on image data output from the panning CCD device 23. In this example, image data for displaying a virtual image such as a liquid crystal display 5 or a CRT device and image indicating data for displaying a virtual image such as a keyboard 1 'are prepared in the image processing apparatus 3 in advance.

A display 24 is connected to the image processing apparatus 3, and the touch panel 1 recognized by the position detection mechanism 4 is displayed. In this example, a stereoscopic virtual image of the keyboard 1 'having an arbitrary key arrangement is present at a position to which the information input surface of the touch panel 1 belongs in the real space.

The special glasstron 2 shown in FIG. 5 constitutes a non-transmissive head-mounted display, and includes a normal CCD image pickup device 25 and the above-mentioned panning CCD device 2.
3 and a liquid crystal display device for right eye display (hereinafter referred to as LCD)
26 and an LCD 27 for left-eye display.

That is, a normal CCD image pickup device 25 and a panning CCD device 23 are arranged side by side at a position corresponding to the operator's eyebrow, and an external image to which the operator belongs is taken by the former, and a touch panel is taken by the latter. One of the four light emitting diodes LED1 to LED4 is panned and shot. Therefore, when the operator looks at the touch panel 1 with the reference plane setting function, the panning CC is performed in the direction of the reference plane.
The D device 23 is turned.

An LCD 26 is attached to the special glasstron 2 at a position facing the right eye of the operator. For example, a touch panel 1 of the operator photographed by a normal CCD imaging device 25 and a keyboard 1 prepared in advance One of the stereo image obtained by synthesizing the virtual image of 'and the virtual image of the liquid crystal display 5 is displayed. An LCD 27 is attached at a position facing the left eye of the operator, and the other of the stereo image obtained by synthesizing the touch panel 1, the virtual image of the keyboard 1 ', and the virtual image of the liquid crystal display 5 is displayed. The special glasstron 2 is mounted on the face or the head of the operator, and the stereo image of the LCD 26 and the stereo image of the LCD 27 are guided to the operator's eyeball. As a result, the touch panel 1 as the background image to which the operator belongs, the virtual image on the liquid crystal display 5, and the virtual image on the keyboard 1 'are synthesized in the head.

The special glasstron 20 shown in FIG. 6 constitutes a transmissive head-mounted display, and does not include a normal CCD image pickup device 25. Accordingly, the transmissive head-mounted display has a panning CCD device 23, a liquid crystal shutter 28 for capturing an external image, and an LCD 29 as an image display element.

For example, a panning CCD device 23 is disposed at a position corresponding to the eyebrows of the operator. When the operator looks at the touch panel 1 having the reference plane setting function, the four light emitting diodes of the touch panel 1 LED1 to LED4
Is panned. A liquid crystal shutter 28 is provided at a position corresponding to the left and right eyes of the operator.
When the liquid crystal shutter 28 is opened, the liquid crystal shutter 28
The real image of the touch panel 1 placed in front of the operator who has passed through is guided directly to the eyeball.

An LCD 29 is attached to a portion of the special glasstron 2 located beside the left or right eye of the operator, and a virtual image of the keyboard 1 'and a liquid crystal display are provided in the same manner as the special glasstron 2 described above. 5 virtual images are displayed. Although not shown, a liquid crystal shutter 28 and
Optical means such as a polarizing beam splitter is provided between the LCD 29 and a real image of the operator's touch panel 1 and a virtual image of the keyboard 1 ′ and a virtual image of the liquid crystal display 5 are guided to the operator's eyeball. ing. Thereby, the touch panel 1 as a background image to which the operator belongs
And the virtual image of the liquid crystal display 5 and the keyboard 1 ′
Is synthesized in the head with the virtual image.

Next, the internal configuration of the panning CCD device 23 of the interline transfer system will be described. The panning CCD device 23 shown in FIG.
On the substrate 31, photodiodes PHij (i = 1 to n, j = 1 to 1) as photoelectric conversion elements forming one pixel are provided.
m) are arranged in a matrix of n columns × m rows.

In the column direction of the substrate, m
Vertical transfer units 32 are provided, and the photodiodes PH
The signal charge read from ij is transferred in the vertical direction (follow-up direction) based on the vertical read signal S1. The vertical transfer unit 32 is connected to the horizontal transfer unit 33, and the signal charges are transferred in the horizontal direction based on the horizontal readout signal S2. In this example, the signal charge is read out from the photodiode PHij to the vertical transfer unit 32 at least a plurality of times during the same field period in order to perform a panning shot.

The panning CCD device 23 has a fisheye lens 35 shown in FIG. The fisheye lens 35 is provided on the optical axis of the CCD image sensor 36, for example. The fish-eye lens 35 can image the touch panel 1 and the like with the reference plane setting function of the operator in a wide range. Of course, a normal lens may be used, but since the field of view is narrowed, the operator must tilt the head more toward the touch panel 1.

Next, the circuit configuration of the virtual image three-dimensional composition device 100 will be described. The virtual image three-dimensional composition device 100 shown in FIG. 9 is roughly composed of three circuit blocks. The first circuit block is a touch panel 1 having a reference plane setting function, and includes a touch switch array 6, a jog dial 12, a blink control circuit 13, and a control system therefor.

The control system of the touch panel 1 is an internal bus 1
Eight. The internal bus 18 has an interface (I
/ O) 8, 9, the transmission unit 14, the CPU 15, the ROM 16, and the RAM 17 are connected. The interface 8 is connected to the jog dial 12 described above, and operation information by an operator is input to a control system. Further, an interface 9 is connected to the touch switch array 6, and contact position information based on the position touched by the operator is input to the control system.

Further, a ROM 16 is connected to the internal bus 18 and stores a system program for controlling the touch panel 1 and control information such as a memory read procedure. A working RAM is provided on the internal bus 18.
17 is connected, and system programs, contact position information by the touch switch array 6, and operation information by the jog dial 12 are temporarily recorded.

The transmission section 14 is connected to the internal bus 18, and the touch plate section 1 read from the RAM 17 is connected to the transmission section 14.
1 and the operation information by the jog dial 12 are transmitted after being modulated into a predetermined transmission signal. In this example, a wireless transmitter is used for the transmission unit 14. Of course, an infrared light emitter may be used for the transmission unit 14. The CPU 15 is connected to the internal bus 18, and controls input / output of the interfaces 8 and 9, the ROM 16, the RAM 17, and the transmission unit 14.

The second circuit block is a special glasstron 2
In the case of the non-transmissive type, the panning CCD device 23 described above, the normal CCD image capturing device 25, and the LC for displaying the right eye are used.
D26 and LCD 27 for left eye display.

The third circuit block is the image processing device 3 and has an internal bus 41. The internal bus 41 includes an interface (I / O) 42, an image capture unit 43, an image processing unit 44, a CPU 45, a ROM 46, a RAM 47,
An E ² PROM (electrically writable and erasable read-only memory) 48 and a receiving unit 49 are connected. Panning CCD device 23, normal CCD imaging device 2
5. LCD 26 for right eye display and LCD 2 for left eye display
7 is connected to the internal bus 41 via the interface 42.

The internal bus 41 has E ² as a storage medium.
The PROM 48 is connected, and stores an algorithm for stereoscopically combining the image of the virtual body with the external image to which the operator belongs. In this example, at least an algorithm that recognizes the touch panel 1 that can be imaged in the real space to which the operator belongs and combines a virtual image of an arbitrary image display device with an image of the recognized touch panel 1 in the virtual space is used. Is stored. In this example, image data for displaying a virtual image of an image display device such as the keyboard 1 ', the liquid crystal display 5, or a CRT device is stored in the E ² PROM 48 in advance.

When such an algorithm and image data are stored in the E ² PROM 48, for example, when controlling the virtual image three-dimensional composition device 100, the algorithm is read out from the E ² PROM 48 and belongs to the operator. When the touch panel 1 that can be imaged in the real space is recognized, the keyboard 1 ′ and the liquid crystal display 5 are added to the recognized virtual image or real image of the touch panel 1 based on the image data read from the E ² PROM 48. Or CR
A virtual image of an image display device such as a T device or a 3D polygon of a character such as a snowman is synthesized.

Accordingly, the touch panel 1 operated by the operator
The virtual image processing can be performed with good reproducibility, such that the virtual image of the keyboard 1 'and the virtual image of the liquid crystal display 5 are stereoscopically combined with the external image of the outside world to execute the image processing. The virtual image stereoscopic synthesis device 100 such as a processing device can be configured with high reproducibility.

Further, a ROM 46 is connected to the internal bus 41, and stores a system program for controlling the virtual image stereoscopic image synthesizing apparatus 100, control information such as a memory reading procedure, and the like. A working RAM 47 is connected to the internal bus 41, and a system program,
Display information for displaying a virtual image such as the keyboard 1 ', the liquid crystal display 5, or the CRT device is temporarily recorded.

A receiving section 49 is connected to the internal bus 41 to receive a transmission signal transmitted from the transmitting section 14 of the touch panel 1. Thereafter, the transmission signal is transmitted to the touch plate section 11 for contact position information, a jog dial, and the like. The operation information is demodulated into the operation information. The demodulated contact position information and operation information are temporarily stored in the RAM 47. A radio receiver is used for the receiving unit 49. Of course, an infrared receiver may be used for the receiver 49.

A CPU 45 is connected to the internal bus 41, and an interface 42, an image capture unit 43, an image processing unit 44, a ROM 46, a RAM 47, an E ² PROM 4
8 and the control of the input / output of the receiving unit 49, the panning CCD device 23, the CCD imaging device 25, the LCD 26 and the LCD 2
7 is controlled. For example, the contact position information and the operation information are read from the RAM 47 and the keyboard 1 ′,
Display control for changing a virtual image such as the liquid crystal display 5 or a CRT device is performed.

The interface 42 has an image processing unit 4
For example, the normal images of the four light emitting diodes LED1 to LED4 of the touch panel 1 shown in FIG. 10 captured by the normal CCD image pickup device 25 are transmitted to the image processing unit via the interface 42 together with the control instruction of the CPU 45. 44, where predetermined image processing is performed.
Again, the special glasstron 2 via the interface 42
The data is transferred to the LCD 26 and the LCD 27 and the like.

An image capture unit 43 is connected to the interface 42. Upon receiving a control command from the CPU 45, a predetermined capture process for acquiring image data of a blinking pattern input from the panning CCD device 23 is performed. The image data of this blinking pattern is expressed as a change in luminance corresponding to the passage of time. An image processing unit 44 as a calculating means is connected to the image capturing unit 43,
With respect to the image data on which the predetermined image processing has been performed, the synchronization deviation of the blinking pattern is corrected, or the reference surface of the touch panel 1 to which the operator belongs is obtained.

For example, in the image processing unit 44, the panning CC
Panning signal (luminance signal) S output from D device 23
The blinking pattern of OUT is converted into a spatial arrangement pattern on the XY plane including the four follow shot bright points P1 to P4 within the image area defined by the window W shown in FIG. Then, by scanning the arrangement pattern, at least the position coordinates (X
1, Y1), (X2, Y2), (X3, Y3), (X
4, Y4) are required. These four bright points P1 to P4 are four light emitting diodes LED1 to LED4 of the touch panel 1 placed in front of the operator. The position coordinates of the four light emitting diodes LED1 to LED4 in the real space are known, and the position coordinates are (x1, y1), (x2, y2),
(X3, y3) and (x4, y4).

Therefore, the touch panel 1 in the real space described above
Is obtained by calculating a conversion matrix projected onto the mounting positions of the four light emitting diodes LED1 to LED4. Here, a point (xi, yi, 0) on the plane of the real space
Is moved by a certain translation / rotational motion, and a point projected on the image coordinate system by the perspective transformation is represented by (Xi, Yi).
There is a relationship between the two as expressed by equation (1).

[0067]

(Equation 1)

Where a1... A8 are unknown coefficients and C1
External parameters (position and direction) such as CD imaging device 25
And internal parameters such as focal length. These parameters are the position coordinates (x1,
y1), (x2, y2), (x3, y3), (x4, y
4) and four sets of position coordinates (X1, Y1), (Y2, Y2), (X3, Y3),
If (X4, Y4) exists, it can be obtained by solving the equation (2).

[0069]

(Equation 2)

The position coordinates of the four points obtained here (X1, Y
1), (X2, Y2), (X3, Y3), (X4, Y
4), the reference surface of the touch panel 1 in the real space shown in FIG. 10 is recognized.

Specifically, on the arrangement pattern shown in FIG. 11, when the moving image pickup direction is the Y axis and the direction orthogonal to the Y axis is the X axis, the image processing unit 44 sets the same direction as the moving image pickup direction. Alternatively, the luminance signal values are added in the opposite direction. When the added value is plotted on the X axis, four positions where the luminance signal value plotted on the X axis is maximum are detected, and the X coordinate values X1 and X corresponding to the four positions are detected.
2, X3 and X4 are determined. Further, when the acquired image is scanned in the Y direction on the arrangement pattern, among the plurality of luminescent points arranged in the Y direction, the luminescent spot position that first emits light has a Y coordinate value Y1 corresponding to the X coordinate value. , Y2, Y3, Y4
Is required.

Here, the position coordinates of the four light emitting diodes LED1 to LED4 in the real space are defined as wi (i = 1 to 4), and the position coordinates wi of the four light emitting diodes LED1 to LED4 on the camera coordinate system. The expression vector is Ci, and L of the four light emitting diodes LED1 to LED4 is L.
Assuming that the position coordinates on the CD screen are Pi, the rotation matrix of the panning CCD device 23 is R, and the movement vector is T, Expression (3), that is, Ci = R · wi + T (3) Ci = Pi.ki (ki is a scalar). Therefore, the normal CCD imaging device 25
Of the rotation matrix R and its movement vector T,
Since the coordinate conversion between the real space and the virtual space can be easily performed using this as a parameter, an image display device such as a keyboard 1 ′, a liquid crystal display 5 or a CRT device is provided on the reference surface of the touch panel 1 in the virtual space. And a 3D polygon such as a snowman can be synthesized.

Next, the operation of the virtual image three-dimensional image synthesizing apparatus 100 will be described with respect to the position recognition method of the touch panel 1 according to the present invention. In this example, the flashing pattern is different on a predetermined surface of the touch panel 1 having a reference surface setting function capable of imaging.
The two light emitting diodes LED1 to LED4 are mounted, the light emitting diodes LED1 to LED4 are imaged so as to flow in a predetermined imaging direction, and the brightness information of the light emitting diodes LED1 to LED4 imaged here is image-processed to obtain four points of brightness. A point position is obtained, and an external world image to which the operator belongs, a virtual image of the keyboard 1 ′, and a virtual image of the liquid crystal display 5 are three-dimensionally synthesized on the reference surface of the touch panel 1 connecting the bright spot positions in the virtual space. Assume the case.

For example, the operator wears the special glasstron 2 shown in FIG. 5 on his / her head, and firstly images the reference surface of the touch panel 1 on the real space to which the operator belongs in step A1 of the flowchart shown in FIG. In order for the processing system to recognize it,
The operator places the touch panel 1 on a desk or the like, for example, such that the touch plate 11 of the touch panel 1 faces upward. After that, the touch panel 1 is operated and the four light emitting diodes LED1 to LED
4 is supplied with a predetermined voltage and blinks in a predetermined blinking pattern.

In this example, the clock signal C having a predetermined frequency
LK1 is supplied to the light emitting diode LED1 through the resistor R, and a clock signal CLK2 obtained by dividing the clock signal CLK1 by １ ／ is supplied to the light emitting diode LE1 through the resistor R.
D2, the clock signal CLK3 obtained by dividing the frequency of CLK1 by 1/3 is supplied to the light emitting diode LED through the resistor R.
The clock signal CLK4 obtained by dividing the frequency of CLK1 by 1/4 is supplied to the light emitting diode LED4 through the resistor R.
Supplied to

Next, in step A 2, the reference plane of the touch panel 1 in the real space is photographed using the ordinary CCD image pickup device 25, and a stereo image is displayed on the LCD 26 and the LCD 27. On the other hand, a panning CCD device 2
3 is used to pan and shoot the reference surface of the touch panel 1 in the real space. For example, four light emitting diodes LED1 to LED4 attached at positions where a virtual image of the keyboard 1 'and a virtual image of the liquid crystal display 5 are to be combined.
Are blinked so that the blinking pattern is different, so that the blinking pattern is imaged so as to flow in a predetermined imaging direction.

Thereafter, in step A3, image processing is performed to recognize a reference plane arbitrarily set in the real space to which the operator belongs. In the image processing unit 44, for example, a subroutine shown in FIG. 13 is called, and a video capture process is executed in step B1. Then, in step B2, the light emitting diodes LED1 to LED4 at the four corners of the touch panel 1 are recognized. Specifically, the blinking pattern of the luminance signal by the four light emitting diodes LED1 to LED4 captured by the panning CCD device 23 is converted into a spatial arrangement pattern forming an XY plane including the four luminescent points P1 to P4. You.

Thereafter, the arrangement pattern is scanned, and
At least the position coordinates (X1,
Y1), (X2, Y2), (X3, Y3), (X4, Y
4) is calculated, and the above-described equations (1) and (2) are calculated, and the mounting positions of the four light emitting diodes LED1 to LED4 of the touch panel 1 in the real space and the position of the image processing system 4
(X1, Y1), (X2, Y2), (X
(3, Y3) and (X4, Y4) are obtained, and a reference plane is obtained by connecting these four points. Then, in step B3, the image processing unit 44 performs an arithmetic process based on the above-described equation (3), and detects the positional relationship between the panning CCD device 23 and the reference surface of the touch panel 1.

Thereafter, the process returns to step A4 of the main routine of FIG. 12, and the virtual image of the keyboard 1 'and the virtual image of the liquid crystal display 5 are superimposed and synthesized on the reference plane of the virtual space. At this time, in the special glasstron 2 worn by the operator, an external image of the touch panel 1 in a real space by the LCD 26, a virtual image of the keyboard 1 ',
One of the stereo images synthesized with the virtual image on the liquid crystal display 5 is guided to the right eyeball of the operator. L
The other of the stereo images obtained by combining the external image of the touch panel 1 in the real space by the CD 27, the virtual image of the keyboard 1 ', and the virtual image of the liquid crystal display 5 is guided to the left eyeball of the operator.

Therefore, in the real space shown in FIG. 14A, the keyboard 1 ′ and the liquid crystal display 5 are not provided on the information input surface (reference surface) of the touch panel 1.
In the virtual space shown in FIG. 14B, a virtual image of the liquid crystal display 5 can appear on the reference surface of the touch panel 1 and a virtual image of the keyboard 1 ′ can appear on the information input surface of the touch panel 1.

Thus, the background image of the touch panel 1 in the real space to which the operator belongs, and the virtual image of the liquid crystal display 5 and the virtual image of the keyboard 1 ′ appearing in the virtual space are synthesized in the head. The liquid crystal display 5 and the keyboard 1 'can be made to exist as if the reference plane of the touch panel 1 in the real space belongs.

As described above, according to the virtual image three-dimensional composition device 100 of the first embodiment, the four light emitting diodes LED1 to LE attached to the four corners of the touch panel 1
By panning the blinking pattern by D4,
The position of the touch panel 1 can be easily recognized by the image processing system from the positions of the four light emitting diodes LED1 to LED4. In addition, the reference surface of the touch panel 1 in the real space to which the operator belongs can be easily recognized by the image processing system with a small amount of calculation.

Accordingly, in the virtual space, an input tool function such as a keyboard 1 ′ having an arbitrary key arrangement is given to the touch panel 1, an arbitrary CRT device as a monitor for the touch panel 1, or a virtual device such as a liquid crystal display 5. It is possible to configure a virtual image processing apparatus or the like that performs image processing by combining images. Further, the calculation load on the image processing unit 44 can be reduced as compared with the conventional method, and the cost of these virtual image processing devices and the like can be reduced.

In this embodiment, since the keyboard, the liquid crystal display and the CRT device are displayed on the display means 24 such as the LCD 26 and the LCD 27 in the special glasstron 2, the desktop CRT device, the notebook size liquid crystal display, A real thing such as a keyboard having a predetermined key layout is not required. Further, the size of the liquid crystal display and the screen of the CRT device in the virtual space can be freely determined. In addition, the key arrangement of the keyboard 1 'to be combined with the touch panel 1 in the virtual space can be freely selected.

(3) Virtual Image 3D Synthesizing Apparatus as Second Embodiment FIG. 15A is a perspective view showing a real image example of the touch panel 1 of the virtual image 3D synthesizing apparatus 200 as the second embodiment, and FIG. Touch panel 1 in the virtual space
FIG. 6 is a perspective view showing an example of combining a virtual image of FIG. In this example, the touch pen 7 and the touch panel 1 as an information input device are provided, and an image drawn on the touch panel 1 by the touch pen 7 is displayed on the liquid crystal display 5 as a virtual image.

Of course, the touch panel 1 shown in FIG.
Are provided at the four corners of the information input surface of FIG. 4 with different blinking patterns, and the light-emitting diodes LED1 to LED4 are imaged by a special glasstron 2 (not shown) so as to flow in a predetermined imaging direction.

In this virtual image three-dimensional synthesizing apparatus 200, FIG.
When a picture of a snowman is drawn on the touch panel 1 by the touch pen 7 shown in FIG. 5A, contact position information forming a line drawing of the snowman is recognized by the control system of the touch panel 1.
Then, contact position information forming a line drawing of the snowman is transmitted from the transmitting unit 14 of the touch panel 1 to the receiving unit 49 of the image processing device 3 of the special glasstron 2 (not shown) (see FIG. 9).

In the special glasstron 2 which has received the contact position information from the touch panel 1, the following image processing is performed to synthesize a line drawing of a snowman with the virtual image of the liquid crystal display 5 shown in FIG. 15B. For example, the reception unit 49 shown in FIG. 9 receives a transmission signal transmitted from the transmission unit 14 of the touch panel 1, and then demodulates contact position information forming a line drawing of a snowman from the transmission signal. The contact position information after demodulation is temporarily stored in the RAM 47.

This contact position information is stored in the
LCD of special glasstron 2 read from AM47
Display control is performed to change the display screen of the liquid crystal display 5 which is a virtual image displayed on the LCD 26 and the LCD 27. This display control is embodied by the image processing unit 44 executing image processing for fitting a line drawing of a snowman on the display screen of the liquid crystal display 5 which is a virtual image according to a control command from the CPU 45. As a result of this display control, a line drawing of a snowman can be combined with the virtual image on the liquid crystal display 5 shown in FIG. 15B.

As described above, according to the virtual image three-dimensional composition device 200 as the second embodiment, the position of the touch panel 1 can be easily recognized by the image processing system from the positions of the four light emitting diodes LED1 to LED4. Therefore, a virtual image of the keyboard 1 ′ is synthesized and displayed on the information input surface of the touch panel 1 in the virtual space, or the keyboard 1 ′ is displayed.
The liquid crystal display 5 which is a virtual image is displayed on the top,
Image processing such as synthesizing a virtual image such as a snowman on the liquid crystal display 5 can be performed with good reproducibility. Also in this embodiment, since the liquid crystal display 5 is displayed on the display means 24 in the special glasstron 2, there is no need for a desktop CRT device or a notebook-size real liquid crystal display.

(4) Three-Dimensional Virtual Image Synthesizing Apparatus as Third Embodiment FIG. 16A is a perspective view showing a real image example of the touch panel 1 of the three-dimensional virtual image synthesizing apparatus 300 according to the third embodiment, and FIG. FIG. 4 is a perspective view showing an example of combining virtual images of the touch panel 1, a virtual body, and a processing tool in the virtual space.

In this example, a touch pen 70 having a light emitting diode and a touch panel 1 as an information input device are provided, and a virtual image of an arbitrary character appears on the touch panel 1 in a virtual space or in a peripheral area thereof, and the character is displayed. It is made so that it can be processed by a processing tool.

Of course, the touch panel 1 shown in FIG.
Are provided at the four corners of the information input surface of FIG. 4 with different blinking patterns, and the light-emitting diodes LED1 to LED4 are imaged by a special glasstron 2 (not shown) so as to flow in a predetermined imaging direction.

In the virtual image three-dimensional composition device 300, when the touch panel 1 and the touch pen 70 shown in FIG. 16A are shot by the panning CCD device 23 of the special glasstron 2 (not shown), the keyboard 1 ', the liquid crystal display 5, etc. The 3D polygon 10 such as a snowman prepared in the RAM 47 in the same manner as the image information of FIG.
The image is synthesized and displayed on the CD 27.

Regarding the combined display at this time, as described above, the reference plane connecting the positions of the four light emitting diodes LED1 to LED4 is recognized by the image processing system.
This is embodied by synthesizing a virtual body on the reference plane or on the surrounding area. Then, the light emitting diodes LED5 and LED6 provided on the upper and lower portions of the touch pen 70, respectively.
When the panning is performed, the image is prepared in the RAM 47 in advance in the same manner as the image information of the 3D polygon 10 such as a snowman.
A virtual image such as a spray 71 as a processing tool together with the 3D polygon 10 and the LCD 26 and LC of the special glasstron 2
D27 is superimposed and displayed.

In this case, the reference line connecting the positions of the two light-emitting diodes LED5 and LED6 is recognized by the image processing system in the same manner as in the above-described synthesis processing, and thereafter, on the reference line or in the peripheral area thereof. This is realized by synthesizing a virtual image of the spray 71 as a processing tool. At this time, a paint image that imagines the function of the spray 71 is synthesized.

Therefore, as a result of this synthesizing process, the 3D polygon 10 on the touch panel 1 shown in FIG.
1 can be used to color the snowman. Needless to say, the color material to be ejected from the spray 71 is previously converted into image data for dividing a plurality of colors into the RAM 47 or the E ² PR.
The information is registered and stored in the OM 48 so that the operator can freely select the color type.

The above-mentioned processing tools are not limited to the spray 71, but writing tools such as erasers, pens and colored pencils, and other carpentry tools such as saws, kanazuki, etc. are stored in advance in the RAM 47 or the like as image data. The operator is free to select a processing tool.

As described above, according to the virtual image three-dimensional composition device 300 as the third embodiment, the position of the touch panel 1 can be easily recognized by the image processing system from the positions of the four light emitting diodes LED1 to LED4. Therefore, the 3D polygon 10 appearing on the touch panel 1 in the virtual space
Can be performed with good reproducibility such as coloring by the spray 71.

In this embodiment, since the spray 71 is displayed on the display means 24 in the special glasstron 2, it is needless to say that there is no need for a desktop CRT device or a notebook-size real spray 71. The color of the 3D polygon 10 to be painted in the virtual space can be freely selected. This makes it possible to provide a virtual image processing device such as a “virtual drawing game device” to which the virtual image three-dimensional composition device 300 is applied.

(5) Virtual Image 3D Synthesizing Apparatus as Fourth Embodiment FIG. 17A is a perspective view showing a real image example of a touch panel 1 of a virtual image 3D synthesizing apparatus 400 as a fourth embodiment, and FIG. Touch panel 1 in the virtual space
FIG. 2 is a perspective view showing a display example of the liquid crystal display 5.

In this example, a silicon range finder as a height shape recognizing device and a touch panel 1 as an information input device are provided, and a character obtained in a real space is placed on the touch panel 1 in a virtual space or in a peripheral area thereof. It was made to appear. Of course, FIG.
At four corners of the information input surface of the touch panel 1 shown in FIG.
D4, the light emitting diodes LED1 to LED
4 is imaged by a special glasstron 2 (not shown) so as to flow in a predetermined imaging direction.

The virtual image three-dimensional synthesizing apparatus 400 has the configuration shown in FIG.
7A, a silicon range finder 50 shown in FIG.
A three-dimensional height shape in a real space of a character such as a snowman is measured. In this example, the three-dimensional height shape information of the target object acquired by the silicon range finder 50 is supplied to the touch panel 1 and passed through the control system of the touch panel 1 to the image processing device 3 of the special glasstron 2 (not shown).
Transferred to Of course, the three-dimensional height shape information may be directly transferred from the silicon range finder 50 to the image processing device 3.

This silicon range finder 50 is a device for measuring a three-dimensional height shape from a preset reference plane to an arbitrary target object, and emits light to be measured for irradiating a target object such as a snowman. A laser light source to be generated,
Scanning means for shaping the light to be measured by the laser light source into slit light and scanning the target object, and a light receiving element for detecting return light from the target object for each pixel.

In this virtual image three-dimensional synthesizing apparatus 400, FIG.
When the real snowman 10 'is placed on the silicon range finder 50 shown in FIG. 7A, the three-dimensional height of the snowman 10' in the real space from the light receiving element is scanned by scanning the sheet-like laser light on the snowman 10 '. The shape information is obtained.

This three-dimensional height shape information is stored in the touch panel 1
Is recognized by the control system. Then, the three-dimensional height shape information forming the three-dimensional shape of the snowman 10 'is transmitted from the transmission unit 14 of the touch panel 1 (not shown) to the reception unit 49 of the image processing device 3 of the special glasstron 2 (see FIG. 9).

In the special glasstron 2 which has received the three-dimensional height shape information from the touch panel 1, the virtual image on the liquid crystal display 5 shown in FIG.
The following image processing is performed to synthesize the D (2D) polygon 10. For example, the reception unit 49 shown in FIG. 9 receives a transmission signal transmitted from the transmission unit 14 of the touch panel 1 and then demodulates three-dimensional height shape information forming a three-dimensional shape of a snowman from the transmission signal. The three-dimensional height shape information after demodulation is temporarily stored in the RAM 47.

The three-dimensional height shape information is read from the RAM 47 by the CPU 45, and is subjected to display control for changing the display screen of the liquid crystal display 5 which is a virtual image displayed on the LCD 26 and the LCD 27 of the special glasstron 2. Will be applied. In this display control, the image processing unit 44 displays a 3D image of the snowman 10 ′ on the display screen of the liquid crystal display 5 as a virtual image according to a control command from the CPU 45.
(2D) This is embodied by executing image processing for fitting the polygon 10. As a result of this display control, FIG.
A virtual image of the snowman 10 ′ acquired by the silicon range finder 50 is superimposed on the virtual image of the liquid crystal display 5 shown in FIG.

As described above, according to the virtual image three-dimensional composition device 400 as the fourth embodiment, the position of the touch panel 1 can be easily recognized by the image processing system from the positions of the four light emitting diodes LED1 to LED4. Therefore, the liquid crystal display 5 which is a virtual image is displayed on the information input surface of the touch panel 1 in the virtual space, and the virtual image such as the snowman 10 ′ acquired by the silicon range finder 50 is synthesized on the liquid crystal display 5. It is possible to perform a virtual image synthesizing process with good reproducibility.

Also, in this embodiment, since the liquid crystal display 5 is displayed on the display means 24 in the special glasstron 2, a CRT device on a desktop or a real liquid crystal display of a notebook size is not required. The size of the liquid crystal display in the virtual space can be freely determined.

(6) Three-Dimensional Virtual Image Synthesizing Apparatus as Fifth Embodiment FIG. 18A is a perspective view showing a real image example of a two-hand operation type input tool 51 of a three-dimensional virtual image synthesizing apparatus 500 as a fifth embodiment. FIG. 18B is a perspective view showing a display example of the input tool 51 and the liquid crystal display 5 in the virtual space.

In this example, a two-handed input tool 51 is provided as an information input device, and appears on the input tool 51 in the virtual space or in a peripheral area thereof based on instruction information for a character operated in the real space. This is to control the movement of the character that has been made to move.

Of course, different positions of the two-handed input tool 51 shown in FIG.
Two light emitting diodes LED1 to LED3 are provided, and the light emitting diodes LED1 to LED3 are imaged by a special glasstron 2 (not shown) so as to flow in a predetermined imaging direction.

This virtual image three-dimensional synthesizing apparatus 500 has the configuration shown in FIG.
A two-handed input tool 51 in which the handle shapes shown in FIG. 8A are arranged symmetrically is provided, and constitutes a character controller such as a PlayStation (Sony: registered trademark). The main body of the input tool 51 is divided into a left-hand operation part 57, a right-hand operation part 58, and an intermediate part 59. Two operation buttons 52 and 53 are provided in the left hand operation unit 57. When the operation button 52 or the operation button 53 is pressed with the left thumb, character instruction information for controlling the movement of the character on the display screen is generated. Is done.

A jog dial 54 is provided on the side of the left-hand operation section 57, and outputs control information other than character instruction information. Two operation buttons 55 and 56 are also provided on the right hand operation unit 58, and other character instruction information for controlling the movement of the character on the display screen by pressing the operation buttons 55 and 56 with the right thumb. Is generated. The middle part 59 is connected between the left hand operation part 57 and the right hand operation part 58. Middle part 5
9 is provided with the flicker control circuit 13 described above.

In this example, the light emitting diode LED1 is provided at the upper end of the left hand operation section 57,
3 is provided at the upper end of the right hand operation unit 58, and the light emitting diode LED2 is provided at the intermediate part 59.

In this virtual image three-dimensional synthesizing apparatus 500, FIG.
By pressing the operation button 52, 53, 55 or 56 shown in FIG. 8A, character instruction information is output from the two-hand operation type input tool 51 to the special glasstron 2 (not shown). This character instruction information is output to an image processing device 3 such as a special glasstron 2 by a wireless method or a wired method.

In the special glasstron 2 which receives the character instruction information from the two-hand operation type input tool 51,
The following image processing is performed to control the movement of the character on the virtual screen of the liquid crystal display 5 shown in FIG. 18B. For example, based on the character instruction information transmitted from the two-hand operation type input tool 51, the character is moved to the left, the character is jumped, or the character is held, for example, fired from a weapon or the like toward the opponent character. Image processing for synthesizing and displaying a virtual image such as a bullet is executed. As a result of this display control, FIG.
The movement of the character can be controlled on the liquid crystal display 5 shown in FIG.

As described above, according to the virtual image three-dimensional composition device 500 as the fifth embodiment, the position of the two-handed input tool 51 can be easily determined by the image processing system from the positions of the four light emitting diodes LED1 to LED3. Since it can be recognized, the liquid crystal display 5 which is a virtual image is displayed adjacent to the two-hand operation type input tool 51 in the virtual space, and an image which controls the movement of the character on the liquid crystal display 5 is displayed. Processing can be performed with good reproducibility.

Also in this embodiment, since the liquid crystal display 5 is displayed on the display means 24 in the special glasstron 2,
This eliminates the need for a desktop CRT device or a real notebook-sized liquid crystal display. The size of the screen of the liquid crystal display 5 in the virtual space can be freely determined.

In each of the embodiments described above, the touch panel 1 and the two-hand operation type input tool 51 have been described with respect to the information input device. However, the present invention is not limited to this. Any type of information input device can be used because it can be recognized.

(7) Three-Dimensional Virtual Image Synthesizing Apparatus as Sixth Embodiment FIG. 19A shows a real image of a touch panel 1 and a touch pen 80 with a light emitting diode and a push button of a virtual image three-dimensional synthesizing apparatus 600 as a sixth embodiment. FIG. 19B is an example.
FIG. 3 is an image diagram showing a synthesis example of the 3D polygon 10 and the virtual tweezers 82 on the touch panel 1 in the virtual space.

In this example, a touch pen 80 having a light emitting diode and a push button and a touch panel 1 as an information input device are provided, and a snowman (3D polygon 10) that images an arbitrary character on the touch panel 1 in a virtual space or in a peripheral area thereof. And the like, and the state of the 3D polygon 10 is changed by virtual tweezers (hereinafter simply referred to as tweezers) 82 as a processing tool.

In order to realize the "pinching function" of the tweezers 82, a push button 81 is provided on the touch pen 80, and when the push button 81 is pressed in a real space,
The tweezers 81 in the virtual space can be closed and opened. In this example, a vibration function is added to the touch pen 80. This vibrating function functions as a sensor when the tip of the tweezers 82 touches the 3D polygon 10 in the virtual space.

Of course, the touch panel 1 shown in FIG.
Are provided at the four corners of the information input surface of FIG. 4 with different blinking patterns, and the light-emitting diodes LED1 to LED4 are imaged by a special glasstron 2 (not shown) so as to flow in a predetermined imaging direction.

In this virtual image three-dimensional synthesizing device 600, when the touch panel 1 and the touch pen 80 shown in FIG. 19A are panned by the panning CCD device 23 of the special glasstron 2 (not shown), the same as in the first embodiment. The 3D polygon 10 such as a snowman prepared in the RAM 47 and the touch panel 1 together with the LC of the special glasstron 2
D26 and LCD27 are combined and displayed.

At this time, as described above, the touch panel 1
The reference plane connecting the positions of the four light-emitting diodes LED1 to LED4 is recognized by the image processing system, and then a virtual body is synthesized on the reference plane or in the peripheral area thereof. Then, when the light emitting diodes LED5 and LED6 provided on the upper and lower portions of the touch pen 80 are shot in a panning manner,
A virtual image of the tweezers 82 shown in FIG. 19B prepared in the RAM 47 together with the 3D polygon 10 is synthesized and displayed on the LCD 26 and the LCD 27 of the special glasstron 2 in the same manner as the image information of the 3D polygon 10 such as a snowman in advance.

Further, the “pinching function” of the tweezers 82
By pressing the push button 81 of the touch pen 80,
For example, the blinking pattern of the light emitting diode LED5 or LED6 is changed, and the change in the blinking pattern is recognized by the image processing system. The tweezers 82 can be closed or opened by using the change point of the blinking pattern as a trigger.

FIGS. 20A to 20D are image diagrams showing display examples of the 3D polygon 10 by the virtual image three-dimensional composition device 600. FIG. 20A is a display example in which tweezers are brought close to the 3D polygon 10, and FIG. 20B is a 3D polygon 10
6 is a display example in which tweezers are brought into contact with. This FIG. 20B
In this case, when the tip of the tweezers 82 touches the 3D polygon 10, the vibrating function of the touch pen 80 causes the vibrator to vibrate.

[0130] The observer who feels the vibration can use the touch pen 8
When the 0 push button 81 is pressed, the tweezers 82 can be closed. When the tweezers 82 are further advanced to the 3D polygon 10 shown in FIG. 20C with the tweezers 82 closed, the 3D polygon 10 can be dented. Alternatively, when the push button 81 of the contact touch pen 80 is pressed in a state where the tweezers 82 bite into the 3D polygon 10, a part of the 3D polygon 10 shown in FIG. 20D is pulled by the tweezers 82 in a state where the tweezers 82 are closed. be able to.

As described above, according to the virtual image three-dimensional composition device 600 as the sixth embodiment, the position of the touch panel 1 can be easily recognized by the image processing system from the positions of the four light emitting diodes LED1 to LED4. Therefore, the 3D polygon 10 appearing on the touch panel 1 in the virtual space
Can be constructed by pushing or pulling the device with tweezers 82 to play a pet.

(8) Other Information Input Device FIG. 21 is a perspective view showing a configuration example of another information input device 40 according to the present embodiment. In this example, a three-dimensional shape measuring mechanism is constituted by the liquid crystal shutter and the trackball.

For example, the information input device 40 has a housing 91 shown in FIG. The housing 91 has a keyboard 9
2, key information is input. At the four corners of the keyboard mounting surface, light emitting diodes LE for setting a reference surface are provided.
D1 to LED4 are provided, and the panning is performed by the panning CCD device 23 as described in the first embodiment. Of course, the light emitting diodes LED1 to LED at the four corners
4 are (x1, y1), (x2, y2), (x
3, y3) and (x4, y4). The four light emitting diodes LED1 to LED4 are a blink control circuit 1
The blinking control is performed so that the blinking pattern differs depending on the number 3.

The housing 91 has a hinge 94. A liquid crystal shutter 93 is movably attached to the hinge 94 so that the sheet-like slit light L0 can scan. Further, a trackball 95 made of a transparent member is provided below the keyboard mounting surface, and a silicon range finder (not shown) is mounted in the trackball 95. This trackball 9
Reference numeral 5 denotes a motor that is rotated around the x, y, and z axes by a motor (not shown). The liquid crystal shutter 93 and the silicon range finder in the trackball 95 constitute a three-dimensional shape measuring mechanism 90.

That is, when the slit light L0 emitted by opening and closing the liquid crystal shutter 93 is reflected by an arbitrary target object 30 on the keyboard 92, the return light reaches the trackball 95. In the silicon range finder in the trackball, return light reflected from the target object 30 is detected for each pixel Pij. Here, the pixel Pij
Is any position (i, j) on the silicon range finder
Indicates a pixel that exists in.

In this type of silicon range finder,
The timing of the return light passing through each pixel Pij is measured, and when a reference plane in the silicon range finder is set, the reference plane and the slit light L0 that are the emission angles of the slit light L0 with the liquid crystal shutter 93 opened are set. Are obtained. The angle between the reference plane and the line of sight of the pixel Pij is βij, the offset distance between the reference plane and the liquid crystal shutter 93 is A, and the pixel Pij
Assuming that the offset distance between ij and the liquid crystal shutter 93 is Bij, the three-dimensional height information Zij between the reference plane and the target object 30 is obtained by equation (4).

[0137]

(Equation 3)

Thus, the three-dimensional height shape from the reference plane to the target object 30 can be measured in parallel and in real time for all the pixels in the silicon range finder by only one scan of the slit light L0. . In this example, a CCD device (motion eye) (not shown) is mounted on a partial area of the silicon range finder in the trackball, the color of the object 30 is imaged, and the color information is output.

As described above, according to the other information input device 40 of the present embodiment, the four light emitting diodes LED1 to LED4 for setting the reference plane and the three-dimensional shape measuring mechanism 90 are provided. Is moved around the x, y, and z axes by the motor. When the slit light L0 emitted from the liquid crystal shutter 93 hits the object 30 and returns, the return light is reflected at multiple angles by the silicon range finder in the trackball 95. Can be obtained.

Therefore, the target object 3 such as a real snowman
0 can be imaged over a wide area,
Image data on the 3D polygon 10 can be acquired in a wide range. In addition, the 3D
When the image of the polygon 10 is three-dimensionally synthesized, the light-emitting diodes LED1 to LED4 at the four corners of the information input surface are shot in a panning manner, so that the reference plane in the real space to which the observer belongs is performed in the same manner as in the first embodiment. Can be easily recognized with a small amount of calculation.

(9) Three-Dimensional Virtual Image Synthesizing Apparatus as Seventh Embodiment FIG. 22A is an example of a real image of the information input device of the three-dimensional virtual image synthesizing apparatus 700 according to the seventh embodiment.
FIG. 4 is an image diagram showing a synthesis example of a liquid crystal display 5 in a virtual space.

In this example, an information input device 40 having a three-dimensional shape measuring mechanism 90 is provided so that a character acquired in a real space appears on the information input device 40 in a virtual space or in a peripheral area thereof. It was done. Of course, four light emitting diodes LED1 to LED4 having different blinking patterns are provided at four corners of the information input surface of the information input device 40 shown in FIG.
An image is taken by a special glasstron 2 (not shown) so that the ED1 to LED4 flow in a predetermined imaging direction.

This virtual image three-dimensional synthesizing apparatus 700 has the configuration shown in FIG.
A three-dimensional shape measuring mechanism 90 shown in FIG. 2A is provided, and a three-dimensional height shape in a real space of a character such as a snowman is measured. In this example, the three-dimensional height shape information of the target object 30 acquired by the silicon range finder in the trackball 95 is processed in the information input device 40 and transferred to the image processing device 3 of the special glasstron 2 (not shown). .

In this virtual image three-dimensional synthesizing apparatus 700, FIG.
When a real snowman 10 ′ is placed on a keyboard 92 of the information input device 40 shown in FIG. 2A, a sheet-like laser beam is scanned by the liquid crystal shutter 93 over the snowman 10 ′, thereby causing a silicon range in the trackball 95 to be scanned. The three-dimensional height shape information of the snowman 10 'in the real space can be obtained from the finder.

This three-dimensional height shape information is transmitted to the information input device 4
0 is recognized by the control system. Then, the three-dimensional height shape information forming the three-dimensional shape of the snowman 10 'is transmitted from the transmission unit 14 of the information input device 40 (not shown) to the reception unit 49 of the image processing device 3 of the special glasstron 2 (see FIG. 9). .

In the special glasstron 2 receiving the three-dimensional height shape information from the information input device 40, the special glasstron 2 synthesizes the 3D (2D) polygon 10 of the snowman 10 'with the virtual image of the liquid crystal display 5 shown in FIG. Image processing as described in the fourth embodiment is performed. The three-dimensional height shape information is read from the RAM 47 by the CPU 45 shown in FIG.
Display control is performed to change the display screen of the liquid crystal display 5, which is a virtual image displayed in D27.

The display control is performed by the image processing unit 44 executing image processing for fitting the 3D (2D) polygon 10 of the snowman 10 ′ on the display screen of the liquid crystal display 5 which is a virtual image according to a control command from the CPU 45. It is embodied. As a result of this display control, the virtual image on the liquid crystal display 5 shown in FIG.
The virtual image (3D polygon 10) of the snowman 10 'obtained by the method 0 is synthesized and displayed.

As described above, according to the virtual image three-dimensional composition device 700 as the seventh embodiment, the position of the information input device 40 can be easily recognized by the image processing system from the positions of the four light emitting diodes LED1 to LED4. So you can
The liquid crystal display 5 which is a virtual image is displayed on the information input surface of the information input device 40 in the virtual space, and a virtual image such as a snowman 10 ′ acquired by the three-dimensional shape measurement mechanism 90 is synthesized on the liquid crystal display 5. Can be performed with good reproducibility.

Also in this embodiment, since the liquid crystal display 5 is displayed on the display means 24 in the special glasstron 2, there is no need for a desktop CRT device or a notebook-size real liquid crystal display. The size of the liquid crystal display in the virtual space can be freely determined.

In this embodiment, a case has been described where the snowman 3D polygon 10 appears on the touch panel 1 with the reference plane setting function, on the information input device 40 with the three-dimensional shape measurement function, or on the display screen of the liquid crystal display 5 thereof. , But not limited to, pedestal polygons,
It may be a polygon of light, fire or ice, or even a polygon like armor.

In this embodiment, the case where the non-transmission type special glasstron 2 or the transmission type special glasstron 20 is used has been described. However, the present invention is not limited to this. Of course, a switchable dual-purpose special glasstron may be used.

The virtual image three-dimensional composition device 10 of this embodiment
0 is JP-A-10-123453, JP-A-9-304
727, JP-A-9-304730, JP-A-9-21
1374, JP-A-8-160348, JP-A-8-9
4960, JP-A-7-325265, JP-A-7-2
The present invention can be applied to a transmission type head mounted display described in Japanese Patent No. 70714 and JP-A-7-67055.

In this embodiment, the case where the two-dimensional imaging device of the interline transfer system is used for the panning CCD 23 has been described. However, the present invention is not limited to this, and the case where the two-dimensional imaging device of the frame transfer system is used is described. Even if there is, the same effect can be obtained.

[0154]

As described above, according to the information input device of the present invention, the control means for controlling the blinking patterns of the plurality of light sources attached to the specific positions in the peripheral area of the information input surface so as to be different from each other. It is provided.

With this configuration, when the information input device is imaged by a special photographing device such as a panning CCD device,
The position of the light source can be easily specified as compared with the case where the lighting pattern of the plurality of light sources is controlled not to blink. Therefore, since the position of the information input device can be easily recognized by the image processing system based on the position of the light source, a virtual image stereoscopic device that combines the image of the arbitrary image display device with the information input device in the virtual space is provided. It can be sufficiently applied to a synthesizer or the like.

According to the position recognition device of the information input device of the present invention, with respect to the blinking pattern of the light source imaged so as to flow in the predetermined imaging direction, the luminance information relating to the blinking pattern is processed by image processing. Is provided.

According to this configuration, the position of the light source attached to the information input device can be easily specified.
The position of the information input device can be easily recognized by the image processing system from the position of the light source. Therefore, an arbitrary input tool function such as a keyboard is provided for an information input device in a virtual space, an arbitrary CRT device is used as a monitor for the information input device, or a virtual image stereoscopic synthesis is performed to synthesize a virtual image such as a liquid crystal display. It can be sufficiently applied to devices and the like.

According to the position recognition method of the information input device of the present invention, a plurality of light sources having different blinking patterns attached to a specific surface of the information input device are imaged so as to flow in a predetermined imaging direction. And image processing of the brightness information of the light source obtained to obtain the position of the light source.

According to this configuration, the position of the information input device can be easily recognized by the image processing system from the position of the light source.
Image processing for synthesizing a virtual image such as an image display device can be performed with good reproducibility.

According to the three-dimensional virtual image synthesizing apparatus of the present invention, the above-mentioned information input apparatus, its position recognizing apparatus and its position recognizing method are applied.

With this configuration, an information input device can be provided with an input tool function such as a keyboard having an arbitrary key arrangement in a virtual space, or an arbitrary CRT device or a liquid crystal display can be used as a monitor for the information input device. It is possible to configure a virtual image processing apparatus or the like that can perform image processing by combining virtual images.

According to the storage medium of the present invention, the information input means capable of capturing an image in the real space to which the operator belongs is recognized, and the information input means in the virtual space recognized there is replaced with an arbitrary image display device or the like. An algorithm for synthesizing a virtual image is stored.

With this configuration, it is possible to construct a virtual image three-dimensional synthesizing device that synthesizes a character or a virtual image such as an image display device on an information input device in a virtual space with high reproducibility.

The present invention is very suitable when applied to a position recognition mechanism of an information input tool for recognizing an existing position of a touch panel or the like in an image processing system or the like and a virtual image processing apparatus to which the information input tool is applied.

[Brief description of the drawings]

FIG. 1 is a perspective view showing a configuration example of an information input device 1 as each embodiment according to the present invention.

FIG. 2 is a block diagram showing an example of the internal configuration of the blink control circuit 13.

FIG. 3 is a waveform diagram showing an example of voltage supply to four light emitting diodes LED1 to LED4.

FIG. 4 is a perspective view illustrating a configuration example of a virtual image three-dimensional composition device 100 to which the position recognition device of the information input device according to the first embodiment is applied.

FIG. 5 is a conceptual diagram viewed from the front showing a configuration example of a special glasstron 2 used in each virtual image three-dimensional composition device.

FIG. 6 is a conceptual diagram viewed from the front showing a configuration example of another special glasstron 20 used in each virtual image three-dimensional composition device.

FIG. 7 is a plan view showing an example of the internal configuration of the panning CCD device 23 of the special glasstron 2.

FIG. 8 is a sectional view showing a configuration example of an optical system of the panning CCD device 23.

FIG. 9 is a diagram illustrating an example of a circuit block of the virtual image three-dimensional composition device 100.

FIG. 10 is an image diagram showing an example of a normal image of the touch plate section 11 forming the reference plane.

FIG. 11 is a schematic diagram showing a calculation example of position coordinates of the reference plane.

FIG. 12 is a flowchart of a main routine showing an operation example (part 1) of the virtual image three-dimensional composition device 100.

FIG. 13 is a flowchart of a subroutine showing an operation example (part 2) of the virtual image three-dimensional composition device 100.

14A is an example of a real image of the touch panel 1 in a real space, and FIG. 14B is an image diagram showing an example of a combination of a keyboard 1 ′ and a liquid crystal display 5 in a virtual space.

FIG. 15A is a real image example of the touch panel 1 of the virtual image three-dimensional composition device 200 according to the second embodiment, and FIG.
5 is an image diagram showing a display example of the liquid crystal display 5 in a virtual space.

16A is an example of a real image of the touch panel 1 of the virtual image three-dimensional composition device 300 according to the third embodiment, and FIG.
FIG. 3 is an image diagram showing a synthesis example of a 3D polygon 10 in a virtual space.

17A is an example of a real image of the touch panel 1 of the virtual image three-dimensional composition device 400 according to the fourth embodiment, and FIG.
FIG. 4 is an image diagram showing an example of combining 3D (2D) polygons 10 in a virtual space.

FIG. 18A is a real image example of a two-handed operation input tool 51 of a virtual image three-dimensional composition device 500 as a fifth embodiment, and B is an image diagram showing a composition example of the liquid crystal display 5 in a virtual space. It is.

FIG. 19A is an example of a real image of the touch panel 1 of the virtual image three-dimensional composition device 600 according to the sixth embodiment;
FIG. 3 is an image diagram showing a synthesis example of a 3D polygon 10 in a virtual space.

20A is a display example in which tweezers are brought close to 3D polygon 10, B is a display example in which tweezers are brought into contact with 3D polygon 10, C is a display example in which tweezers are bitten into 3D polygon 10, and D is 3D It is an image figure showing the example of a display which pulled polygon 10 with tweezers.

FIG. 21 is a perspective view illustrating a configuration example of another information input device 40 according to the present embodiment.

FIG. 22A is a real image example of the information input device 40 of the virtual image three-dimensional composition device 700 according to the seventh embodiment;
FIG. 3 is an image diagram showing a synthesis example of a 3D polygon 10 in a virtual space.

[Explanation of symbols]

1 ... touch panel (information input device, information input means), 2, 20 ... special glasstron (synthesis means),
3 image processing device (computing means), 4 position recognition mechanism (position recognition device, position recognition means), 7 touch pen, 10 3D polygon, 11 touch plate section, 13 ... blinking control circuit (control means), 23
· · Panning CCD device (imaging device), 24 · · · display means, 25 · · · CCD imaging device, 26 · · · LCD for right eye display, 27 · · · LCD for left eye display, 32 ·・
・ Vertical transfer unit (charge transfer unit), 33 ... horizontal transfer unit,
50: Silicon range finder (height shape measuring device), 51: Two-hand operation type input tool (information input device), 70: Touch pen with light emitting diode, 71
... spray (processing tool), 80 ... touch pen with light emitting diode & push button, 82 ... tweezers,
90 ... three-dimensional height shape measurement mechanism, 100-700
..Virtual image three-dimensional synthesizing devices, LED1 to LED6...
Light emitting diode (light source)

 ──────────────────────────────────────────────────続 き Continued on the front page F-term (reference) 2C001 BB00 BB10 BC01 BC04 CA00 CA01 CA02 CA08 CA09 CB01 CC00 CC02 CC03 CC06 5B087 AA07 AE00 BC05 BC12 BC13 BC16 BC19 BC26 BC32 CC02 CC12 CC20 CC25 DE00 DJ06 9A001 BB04 DD12 HH28 KKH29 KK62

Claims (23)

[Claims] 1. An image capturing information input device having an information input surface, wherein at least a plurality of light sources attached to specific positions in a peripheral area of the information input surface and input / output of the light source are controlled. An information input device, comprising: a control unit, wherein the control unit performs blink control so that a blink pattern of the light source is different. 2. The information input device according to claim 1, wherein the information input device forms a touch panel that outputs contact position information when the information input surface is touched. 3. The information input device according to claim 1, wherein a keyboard is attached to the information input surface. 4. The information input device according to claim 1, wherein a three-dimensional height shape measuring mechanism is provided on the information input surface. 5. An apparatus for recognizing a position of an information input device in which a plurality of light sources having different blinking patterns are mounted at a specific position, wherein an image pickup means for imaging the light source of the information input device so as to flow in a predetermined imaging direction. A position recognition device for an information input device, comprising: image processing of a luminance signal of the light source by the imaging unit to obtain a position of each of the light sources. 6. A two-dimensional imaging device having a plurality of photoelectric conversion elements constituting each pixel is used as said imaging means, and when transferring signal charges obtained from said photoelectric conversion elements in a predetermined direction. 6. The position recognition device for an information input device according to claim 5, wherein the signal charge is read from the photoelectric conversion element at least a plurality of times during the same field period. 7. The method according to claim 6, wherein the imaging unit and the calculation unit are provided, wherein the calculation unit determines a blinking pattern of a luminance signal by the imaging unit.
6. The image processing apparatus according to claim 5, wherein the image is converted into a spatial arrangement pattern forming an XY plane including a plurality of bright points, and at least a plurality of position coordinates are obtained by scanning the arrangement pattern. Information input device position recognition device. 8. The panning imaging method, wherein a panning imaging direction is set to an Y-axis on an arrangement pattern forming an XY plane including a plurality of bright spots, and an axis orthogonal to the Y-axis is set to an X-axis. The luminance signal values are added in the directions and plotted on the X axis. The position where the luminance signal value plotted on the X axis is the maximum is detected to determine a plurality of X coordinate values. When scanning in the Y-axis direction in (1), among the plurality of bright spots arranged in the panning imaging direction, the position of the bright spot that first emits light is obtained as a Y coordinate value corresponding to the X coordinate value. Item 6. A position recognition device for an information input device according to Item 5. 9. A plurality of light sources having different blinking patterns are mounted on a specific surface of the information input device capable of photographing, and the light sources having different blinking patterns are photographed so as to flow in a predetermined photographing direction. A position information recognizing method for the information input device, wherein the position information of each of the light sources is obtained by performing image processing on the luminance information of the light source. 10. An apparatus for stereoscopically combining an image of a virtual body with an external world image to which an operator belongs, comprising: an image inputting means having an information input surface; A position recognizing unit that recognizes the information input unit; and a synthesizing unit that synthesizes an arbitrary virtual image with an image in a virtual space of the information input unit recognized by the position recognizing unit. At least three or more light sources that are attached at a specific position of the information input means and blink so that the blinking pattern is different, and a panning imaging means for imaging the light sources so as to flow in a predetermined imaging direction, Calculating means for obtaining three positions of the light source by performing image processing on a luminance signal of a blinking pattern by the imaging means, and thereafter obtaining a reference plane by connecting the positions of the three light sources. Virtual image solid synthesizer. 11. A virtual image in which the position recognizing unit and the synthesizing unit are provided, wherein the synthesizing unit images an arbitrary image display device on a virtual space information input unit recognized by the position recognizing unit. The virtual image three-dimensional composition apparatus according to claim 10, wherein the composition is performed. 12. The virtual image three-dimensional composition apparatus according to claim 10, wherein said information input means has a touch panel that outputs contact position information when touching an information input surface. 13. The virtual image three-dimensional composition apparatus according to claim 12, wherein the touch panel is provided, and a touch pen for touching the touch panel is provided. 14. A case where the position recognizing means and the synthesizing means are provided, wherein the synthesizing means is a virtual image which images an arbitrary character on the information input means in the virtual space recognized by the position recognizing means. The virtual image three-dimensional composition apparatus according to claim 10, wherein the composition is performed. 15. A processing tool for processing a character, wherein a virtual image of a character is combined with the information input means on the virtual space, wherein the processing tool has a specific processing tool. 15. The virtual image three-dimensional composition device according to claim 14, wherein at least two or more light sources that blink at different positions and that blink in different positions are provided. 16. When the processing tool and the synthesizing unit are provided, the synthesizing unit is configured to synthesize a virtual image of the function of the processing tool when processing the character. The virtual image three-dimensional composition device according to claim 15, wherein: 17. A height shape recognition device for measuring a three-dimensional height shape of a character in a real space, wherein a virtual image of a character is synthesized in the information input means in the virtual space. The virtual image three-dimensional composition device according to claim 14, wherein the three-dimensional height shape information acquired by the height shape recognition device is supplied to the information input means. 18. A two-dimensional imaging device having a plurality of photoelectric conversion elements constituting each pixel is used as the panning imaging means, and a signal charge obtained from the photoelectric conversion element is transferred in a predetermined direction. 11. The virtual image three-dimensional composition device according to claim 10, wherein the signal charge is read from the photoelectric conversion element at least a plurality of times during the same field period. 19. The method according to claim 19, wherein the imaging unit and the arithmetic unit are provided, wherein the arithmetic unit determines a blinking pattern of a luminance signal by the imaging unit.
By converting into a spatial arrangement pattern forming an XY plane including three luminescent points, scanning the arrangement pattern to obtain at least the position coordinates of three luminescent points, and connecting the position coordinates of the three points The virtual image three-dimensional composition apparatus according to claim 10, wherein the reference plane is recognized. 20. The moving image pickup apparatus according to claim 19, wherein the flow direction is set as an Y axis on an arrangement pattern forming an XY plane including three bright points, and the axis orthogonal to the Y axis is set as an X axis. The luminance signal values are added in the directions and plotted on the X-axis. The position where the luminance signal value plotted on the X-axis is maximum is detected to obtain three X-coordinate values. When scanning in the Y-axis direction in (1), among the plurality of bright spots arranged in the panning imaging direction, the position of the bright spot that first emits light is obtained as a Y coordinate value corresponding to the X coordinate value. Item 13. A virtual image three-dimensional composition device according to item 10. 21. An image pickup means for picking up an external world image to which an operator belongs, and the synthesizing means displays one of a stereo image obtained by synthesizing the external image by the image pickup means and a virtual image prepared by an image display means prepared in advance. A head mounted display having a first image display element and a second image display element for displaying the other of the stereo images, wherein the head mounted display is mounted on a face or a head of an operator; 11. The three-dimensional virtual image according to claim 10, wherein a stereo image by the first image display element and a stereo image by the second image display element are superimposed and guided to an operator's eyeball. Synthesizer. 22. A liquid crystal shutter for opening and closing incident light to capture an external image to which an operator belongs, an image display element for displaying an image of a virtual body to be composited with the external image, An image of the virtual body by the image display element, and a head mounted display having an optical unit that guides an external image to which the operator who has passed through the liquid crystal shutter belongs to the operator's eyeball, The head mounted display includes: When mounted on the face or head of the operator and the liquid crystal shutter is opened, the image of the virtual body by the image display element is superimposed on the external image in real space to which the operator who has passed through the liquid crystal shutter belongs. The virtual image three-dimensional composition device according to claim 10, wherein the virtual image stereoscopic composition device is guided to an operator's eyeball. 23. A recording medium storing an algorithm for stereoscopically combining an image of a virtual body with an external world image to which an operator belongs, wherein at least the recording medium is imaged in a real space to which the operator belongs. A recording medium storing an algorithm for recognizing a possible information input means and synthesizing a virtual image of an arbitrary image display device with an image in the virtual space of the recognized information input means.