JP2000132329A - Device and method for recognizing surface and virtual image solid synthesizer - Google Patents

Abstract

PROBLEM TO BE SOLVED: To pick up an image with best focus so as to move a light source in a prescribed image pickup direction, and to accurately recognize a reference surface based on the position of that light source. SOLUTION: A picture recognizing device 100 is provided with four light emitting diodes LEDi (i=1 to 4), which are attached at the desired positions of an arbitrary object 1, to be flickered so as to make flicker patterns different, panning CCD 23 for picking up images so as to move these light emitting diodes LEDi in the prescribed image pickup direction, arithmetic means 6 for finding the positions of light emitting diodes LEDi by performing image processing to a luminance signal SOUT of this flicker pattern and for finding the reference surface by linking these four light emitting positions later, camera 4 for scrutinized point detection for detecting a scrutinized point(p) of an observer, who gazes at the reference surface, by picking up the image of eyeball motion of that observer, and control means 5 for controlling the optical system of the panning CCD 23 based on a camera output signal S3. In this case, based on the motion of the pupils recognized from the eyeball motion of the observer, a focal distance Sc+Soff between the light emitting diode LEDi, that the observer gazes at, and the panning CCD 23 is corrected.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a surface recognition device, a surface recognition method, and a virtual image three-dimensional synthesis device suitable for application to a three-dimensional display device that synthesizes an image of a virtual body on a reference plane in a virtual space.

More specifically, a control means for adjusting the focal length of a panning imaging means for imaging a plurality of light sources having different blinking patterns is provided, and the control means adjusts the focal length based on the pupil movement recognized from the eye movement of the observer. The focal length between the light source watched by the observer and the imaging means is corrected so that the light source can be imaged so as to flow in a predetermined imaging direction at the best focus, and the reference plane based on the light emission position of the light source Can be recognized with high accuracy.

[0003]

2. Description of the Related Art In recent years, with the improvement of display technology based on virtual reality, a virtual image three-dimensional synthesizing apparatus for providing a viewer with virtual reality over a plurality of image display surfaces has appeared. are doing.

[0004] This type of stereoscopic display device is disclosed in Japanese Unexamined Patent Publication No. 9-23.
7353 can be found in the technical literature. According to this technical document, a projection space of about several meters in length and width is provided, and a display device is arranged on each surface, and a dinosaur,
Images of virtual bodies such as monsters and weapons are displayed in three dimensions. Then, when the observer wears glasses with a liquid crystal shutter and stands in the projection space, it is as if he were at the same place as the virtual object displayed on each display device.

[0005] Further, a weapon that an observer holds in a virtual space is imaged by a camera, and image processing is performed so that a virtual body reacts according to the movement of the weapon. This allows the observer to slip back in time to the primitive era thousands of years ago, and perform dinosaur extermination, etc., as if playing a game.

[0006]

However, according to the conventional three-dimensional display device, the center of gravity of the three-dimensional shape is obtained on the assumption that the projection space has a three-dimensional shape, and the center of gravity of the three-dimensional shape is determined from the center of gravity. The coordinates are determined. Then, based on the relative coordinates, the positional relationship between the camera and the virtual body is obtained, and a virtual body such as a dinosaur is displayed in a composite manner in the virtual space.

For example, when a virtual body such as a dinosaur is to stand on a certain reference plane in the projection space, a rectangular area circumscribing the virtual body is extracted by image processing, and the four corners of the rectangular area are displayed on the image. The position is obtained from the relative coordinates, and the positional relationship between the camera and the rectangular area of the virtual body is obtained using the parameters obtained from the perspective projection conversion method or the like.

[0008] Therefore, a TV program character or the like virtually jumps out on the reference plane of the real space to which the observer belongs,
When trying to construct a virtual image stereoscopic synthesis device that allows the character to play with the palm of the observer in the virtual space, if the conventional system is applied as it is, the stereoscopic display device will become large-scale or the reference plane will be recognized. However, there is a problem that the image processing for the game becomes complicated or the amount of calculation at that time becomes large, which leads to an increase in the cost of a game device or the like.

Therefore, the present invention has been made in view of the above problems, and a reference plane in a real space to which an observer belongs is defined as:
It is an object of the present invention to provide a surface recognition device, a surface recognition method, and a virtual image three-dimensional composition device which can be easily and accurately recognized with the best focus with a small amount of calculation.

[0010]

SUMMARY OF THE INVENTION An object of the present invention is to provide at least three or more light sources attached to a desired position of an arbitrary object and blinking at least in different blinking patterns, and disposing the light sources in a predetermined imaging direction. Panning imaging means for imaging so as to flow, and image processing of a luminance signal of a blinking pattern by the imaging means to obtain three positions of the light source.
Calculating means for obtaining the reference plane by connecting the positions of the light sources of the points; imaging means for detecting the point of interest of the observer who observes the reference plane and detecting the point of interest of the observer; Control means for adjusting the optical system of the imaging means for panning based on the output of the imaging means for detection, the control means,
A plane recognition apparatus characterized in that a focal length between a light source gazed at by the observer and an imaging means for panning is corrected based on the movement of the pupil recognized from the eye movement of the observer. Solved by

According to the surface recognition apparatus of the present invention, since the focal length of the panning imaging means is corrected by the control means based on the output of the gazing point detection imaging means, recognition is made from the eye movement of the observer. The focal length between the light source watched by the observer and the imaging means for panning can be adjusted to an optimal position based on the pupil movement performed.

[0012] Therefore, since the panning imaging means is focused on a specific light source watched by the observer, it is possible to perform imaging with the light source flowing in a predetermined imaging direction at the best focus. Image processing can be optimally performed on the luminance signal of

According to the surface recognition method of the present invention, three or more light sources are attached to desired positions of an arbitrary object, and at least the light sources are blinked so as to have different blink patterns, and the light sources are caused to flow in a predetermined imaging direction. An image is taken by the first imaging system, and the eye movement of the observer watching the light source is taken by the second imaging system, and the observer gazes based on the movement of the pupil recognized from the eye movement of the observer. The focal length between the light source and the first imaging system is corrected, and the luminance signal of the blinking pattern captured by the first imaging system with the corrected focal length is subjected to image processing to determine the positions of three points of the light source. Asked, then
It is characterized in that a reference plane is obtained by connecting the positions of three light sources.

According to the surface recognition method of the present invention, it is possible to focus the panning imaging means on a specific light source watched by the observer, so that the light source is caused to flow in a predetermined imaging direction at the best focus. Can be taken at the same time,
The luminance signal of the blinking pattern can be optimally image-processed. Therefore, since the reference plane can be recognized with high accuracy, this plane recognition method can be sufficiently applied to the plane recognition means of the virtual image stereoscopic synthesis apparatus.

A three-dimensional virtual image synthesizing apparatus according to the present invention is an apparatus for stereoscopically synthesizing an image of a virtual body with an external image to which an observer belongs, and recognizes an arbitrary reference plane in a real space to which the observer belongs. Surface recognition means, display means for displaying the reference plane recognized by the surface recognition means, and synthesis means for synthesizing an image of the virtual body on the reference plane of the virtual space displayed by the display means, The recognizing means includes at least three or more light sources attached to a desired position of an arbitrary object and blinking so as to have different blinking patterns, and a panning imaging for imaging the light sources in a predetermined imaging direction. Means,
Image processing is performed on the luminance signal of the blinking pattern by the imaging means to determine the positions of three points of the light source, and thereafter, calculating means for determining the reference plane by connecting the positions of the three light sources, and an observer who observes the reference plane An image pickup means for detecting a point of gaze of the observer by imaging the eye movement, and a control means for adjusting an optical system of the image pickup means for panning based on an output of the image pickup means for detecting a point of gaze The control means is adapted to correct the focal length between the light source gazed at by the observer and the imaging means for panning based on the movement of the pupil recognized from the eye movement of the observer. It is characterized by having.

According to the virtual image three-dimensional composition device of the present invention,
Since the above-described surface recognition device is applied, it is possible to focus the panning imaging unit on a specific light source watched by the observer. Therefore, it is possible to image the light source in the predetermined imaging direction with the best focus, and
Since the luminance signal of the blinking pattern can be optimally image-processed, the reference plane can be recognized with high accuracy.

[0017]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A surface recognition device, a surface recognition method and a virtual image three-dimensional synthesis device according to embodiments of the present invention will be described below with reference to the drawings. (1) Surface Recognition Device as Embodiment FIG. 1 shows a surface recognition device 10 as each embodiment according to the present invention.
FIG. 2 is a perspective view showing a configuration example of a No. 0. In this embodiment, a control unit for adjusting the focal length of the imaging unit for panning for imaging a plurality of light sources having different blinking patterns is provided,
Based on the movement of the pupil recognized from the eye movement of the observer, the focal length between the light source gazed at by the observer and the imaging means is automatically corrected so that the light source flows in a predetermined imaging direction at the best focus. In addition to this, it is possible to accurately recognize the reference plane based on the light emitting position of the light source.

The surface recognition apparatus 100 of the present invention is provided with a panning imaging means (first imaging system, hereinafter simply referred to as a panning CCD device 23) shown in FIG. She is shot as if shed in the direction. Here, the panning shot means a shooting mode in which signal charges are read out from the photoelectric conversion element (such as a photodiode) a plurality of times during the same field period in the panning CCD device 23.

In this example, four light-emitting diodes LEDi (i = 1 to 4) as a plurality of light sources are provided at a desired position of an arbitrary object 1, for example, a predetermined position of a bracelet for setting a reference plane such as a virtual image three-dimensional synthesizer. 4) is attached, and blinking control is performed so that at least the blinking pattern is different. Here, the virtual image three-dimensional synthesizing device refers to a device that stereoscopically synthesizes a virtual body image with an external image to which an observer belongs in a virtual space.

The arithmetic unit 6 is connected to the panning CCD device 23. The luminance signal SOUT of the blinking pattern obtained from the panning CCD device 23 is image-processed to determine the four light emitting positions of the light emitting diodes LED1 to LED4. After that, a reference plane is obtained by connecting the light emitting positions of the four light emitting diodes LED.

On the other hand, the plane recognition apparatus 100 is provided with an image pickup means (second image pickup system, hereinafter referred to as a gazing point detection camera 4) for detecting a gazing point, and detects an eye movement of an observer gazing at a reference plane. The image is picked up, and the gazing point p of the observer is detected.
Control means 5 is connected to the gazing point detection camera 4 and a camera output signal S3 obtained from the gazing point detection camera 4
, The focal length of the panning CCD device 23 is adjusted. For example, the control means 5 includes the panning CCD device 23
And outputs a zoom control signal S0 to the light emitting diode LED based on the movement of the pupil recognized from the eye movement of the observer.
The focal length between i and the panning CCD device 23 is corrected.

Here, the separation distance from the surface of the observer's eyeball to the gazing point p is denoted by Sh, the focal length from the lens surface of the panning CCD device 23 to the gazing point p is Sc, and the distance from the observer's eyeball to the gazing point p. Assuming that the offset distance between the taking CCD device 23 and the lens surface is Soff, the gazing point p
Is larger than the focal length Sc + Soff, the control means 5 adjusts the optical system of the follow-up CCD device 23 so as to increase the focal length Sc. On the contrary, the separation distance Sh to the gazing point p is the focal length Sc + Sof
If it is less than f, the control means 5 determines that the focal length S
The optical system of the panning CCD device 23 is adjusted so as to shorten c.

FIG. 2 is a flowchart showing an operation example of the surface recognition apparatus 100. In this example, it is assumed that, for example, four light emitting diodes LED1 to LED4 are attached to desired positions of an arbitrary object 1 such as a reference surface setting bracelet in advance. Then, in step A1 of the flowchart shown in FIG. 2, first, the light sources such as the light emitting diodes LED1 to LED4 are controlled to blink so that the blinking patterns are different.

Thereafter, in step A2, the light-emitting diodes LED1 to LED4 are photographed by the panning CCD device 23 so as to flow in a predetermined imaging direction. In parallel with this, in step A3, for example, the eye movement of the observer who gazes at the light emitting diode LED3 is imaged by the gaze point detection camera 4. Thereafter, in step A4, the focal length Sc + Soff between the light emitting diode LED3 and the panning CCD device 23 watched by the observer is corrected based on the pupil movement recognized from the eye movement of the observer.

For example, if the separation distance Sh from the surface of the eyeball of the observer to the gazing point p is greater than the focal length Sc + Soff, the control means 5 controls the panning of the panning CCD device 23 to increase the focal length Sc. The optical system such as the lens and the lens is adjusted. Conversely, when the separation distance Sh to the point of gaze p is smaller than the focal length Sc + Soff, the control unit 5 adjusts the optical system of the panning CCD device 23 so as to shorten the focal length Sc.
Thereby, the light-emitting diode LED3
Distance S between the camera and the optical system of the panning CCD device 23
c + Soff can be automatically corrected.

Thereafter, in step A5, the luminance signal of the blinking pattern imaged by the panning CCD device 23 whose focal length has been automatically corrected is subjected to image processing to determine four light emitting positions of the light emitting diodes LED1 to LED4. Then, in step A6, a reference plane is obtained by connecting the positions of the four light emitting diodes LED1 to LED4.

As described above, according to the surface recognition apparatus 100 of the present embodiment, the follow-up CCD device 23 based on the camera output signal S3 obtained from the gazing point detection camera 4.
Is corrected by the control means 5, so that the light-emitting diode LED3 and the panning CCD device 23 that the observer gazes at the optimal position based on the movement of the pupil recognized from the eye movement of the observer. Can be adjusted to the focal length Sc + Soff.

Therefore, the panning CCD device 23 is optimally focused on the specific light emitting diode LED3 watched by the observer, so that the light emitting diodes LED1 to LED4 are caused to flow in a predetermined imaging direction with the best focus. It is possible to take an image and optimally process the luminance signal SOUT of the blinking pattern. Thus, since the reference plane can be recognized with high accuracy, the plane recognition apparatus 100 can be sufficiently applied to a plane recognition unit of a virtual image stereoscopic synthesis apparatus.

A gyroscope may be provided in the object 1 on which the light emitting diode LED is mounted, and three-dimensional position information based on the posture of the object 1 may be output to the control means 5. Based on the position information by the gyroscope and the camera output signal S3 from the gazing point detection camera 4, the focal length Sc + Soff between the specific light emitting diode LEDi watched by the observer and the panning CCD device 23 with higher precision is adjusted. Can be included.

(2) Three-dimensional virtual image synthesizing apparatus as first embodiment FIG. 3 is a perspective view showing a configuration example of a three-dimensional virtual image synthesizing apparatus 200 according to the first embodiment of the present invention. In this embodiment, a plane recognizing means for recognizing an arbitrary reference plane in the real space to which the observer belongs is provided, and an image of the virtual body is synthesized in the virtual space to which the recognized reference plane belongs. At the position where the reference plane belongs in space, as if
A so-called virtual character breeding game device that allows a character image or the like to jump out and play with the palm of an observer can be configured inexpensively and compactly.

A three-dimensional virtual image synthesizing apparatus 200 shown in FIG. 3 is an apparatus for stereoscopically displaying a virtual body image such as a TV program character on an external image to which an observer belongs. The virtual image three-dimensional synthesis device 200 includes a bracelet 1 for setting a reference plane as an arbitrary object, a special glasstron 2, and an image processing device 3. The bracelet 1 is used, for example, mounted on the left hand of the observer. The main body 21 of the special glasstron 2 is provided with a belt 22. The main body 21 is attached to the face of the observer as if wearing glasses, and the belt 22 extends along the outer periphery of the observer's head. Fixed.

The special glasstron 2 is provided with at least a panning CCD device 23 and display means 24. Normal CC depending on the model of Special Glasstron 2
A D imaging device 25 is provided. The bracelet 1 for setting the reference plane, the panning CCD device 23, and the image processing apparatus 3 constitute a plane recognition unit, and can recognize an arbitrary reference plane in the real space to which the observer belongs.
Of course, the surface recognition device 100 with the automatic correction function described above is applied to this surface recognition means.

When an interline transfer type two-dimensional imaging device having a vertical transfer unit is used as the panning CCD device 23 in this example, a signal is sent from the photoelectric conversion element to the vertical transfer unit a plurality of times during the same field period. The charge is read. When a two-dimensional frame transfer type imaging device having a charge storage unit is used as the panning CCD device 23, signal charges are read from the photoelectric conversion element to the charge storage unit a plurality of times during the same field period.

Further, an image processing device 3 is connected to the special glasstron 2, and image processing for recognizing a reference plane or the like is performed based on image data output from the panning CCD device 23. The display unit 24 is connected to the image processing apparatus 3 and displays the reference plane recognized by the plane recognition unit. The special glasstron 2 may be provided with optical means such as a polarizing beam splitter.
The image of the virtual body is synthesized on the reference plane of the virtual space displayed by 4. In this example, for example, a 3D polygon (snowman) 10 exists as a virtual body at the position to which the reference plane belongs in the real space.

In this example, the reference surface setting bracelet 1
Has a plate portion 11 and a bracelet portion 12 shown in FIG. The plate portion 11 forms a reference surface, and is formed in a flat shape without irregularities. The size of the plate portion 11 is, for example, about the size of a wristwatch, and the length of one side thereof is about 3 cm, and at most about 5 cm. Light emitting diodes (LED1 to LED4) are attached to four corners of the surface of the plate portion 11 as three or more light sources, respectively.
(X1, y1), (x2, y) as coordinates of four points p1 to p4 on the reference plane from which the D polygon 10 is to pop out.
2), (x3, y3) and (x4, y4) are given (corresponding to a reference plane on which an image is to be synthesized in the virtual space).

In this example, four light emitting diodes LED
1-4 are located at each vertex of the rectangle, and the light emitting diodes LE
The distance Lx between D1 and LED3 (x direction) is (x3-x
1), the distance between the light emitting diodes LED2 and LED4 is also Lx, and the distance Lx is (x4-x2).
The distance Ly between the light emitting diodes LED1 and LED2 (in the y direction) is (y2-y1), and the distance between the light emitting diodes LED2 and LED4 is also Ly.
x is (y4-y3). These position information and distance information are registered in advance in a RAM or the like of an image processing system in order to be used for surface recognition.

The four light emitting diodes LED1 to LED4
Is blinked so as to exhibit its function as a mark portion, that is, at least in a different blinking pattern so that its mounting position is clear. This blinking control method will be described with reference to FIG. The light emitted from the light emitting diodes LED1 to LED4 is imaged by the panning CCD device 23 in the special glasstron 2 so as to flow in a predetermined panning imaging direction. This panning is for specifying the reference plane from the mounting positions of the four light emitting diodes LED1 to LED4. The specification of the reference plane is shown in FIGS.
This will be described with reference to FIG.

The bracelet 12 preferably has a structure that is extendable and contractable so as to fit the wrist of the observer. For example,
For example, a Velcro is provided on the band member so that the mounted state can be adjusted. The plate portion 11 is preferably mounted on the back side of the left hand so as to be within the shooting range of the panning CCD device 23.

The special glasstron 2 shown in FIG. 5 constitutes a non-transmissive head-mounted display, and includes a normal CCD image pickup device 25 and the above-mentioned panning CCD device 2.
3, a liquid crystal display device (hereinafter referred to as LCD) 26 for right-eye display as a first image display element, an LCD 27 for left-eye display as a second image display element, and a not-shown gaze point detection It has film-shaped charge-coupled devices (hereinafter referred to as film CCDs) 4R and 4L as the camera 4.

That is, a normal CCD image pickup device 25 and a panning CCD device 23 are arranged side by side at a position corresponding to the eyebrows of the observer, and the former captures an external image to which the observer belongs, and the latter captures a bracelet. One of the four light emitting diodes LED1 to LED4 is panned. Therefore, when the observer looks at the bracelet 1 for setting the reference plane, the panning CCD device 23 is directed toward the reference plane.

In this example, the panning CCD device 23 is provided with an automatic zoom mechanism, and an arbitrary light emitting diode LED
When the user gazes at i, the image processing apparatus 3 can automatically zoom the light emitting diode LEDi.
For this automatic zoom mechanism, for example, the technology of electronic equipment and camera disclosed in Japanese Patent Application Laid-Open No. HEI 8-179193 can be applied.

An LCD 26 is mounted in the special glasstron 2 at a position facing the right eye of the observer. For example, an observer's bracelet 1 photographed by a normal CCD image pickup device 25 and a computer One of the stereo images synthesized with the image of the 3D polygon 10 by graphics (CG) or the like is displayed. Also,
An LCD 27 is attached to a position facing the left eye of the observer, and the other of the bracelet 1 and the stereo image synthesized with the image of the 3D polygon 10 is displayed.

In this example, an LCD 26 for displaying the right eye,
As a camera 4 for detecting the point of gaze, a film CCD 4R shown in FIG.
(Right), 4L (Left) are arranged, and an eye movement of an observer who gazes at the reference plane is imaged to detect a gaze point p of the observer. In this example, the film CCDs 4R and 4L are dispersedly arranged on the LCD 26 and the LCD 27 facing the eyeball of the observer.

In the film CCDs 4R and 4L, FIG.
For example, the image sensor 4A is arranged in a toothless manner (hatched portion) with respect to all the pixels of the matrix of 4 × 6 pixels shown in FIG. Therefore, the image pickup device 4A is not arranged in the white portion, and the images from the LCD 26 and the LCD 27 are passed. The gazing point detection cameras 4R and 4L partially form a silicon film on the film to form a CCD such as a photoelectric conversion element in a toothless shape.
4R and 4L are formed by attaching them in front of the LCD 26 and the LCD 27, for example.

Since the film CCDs 4R and 4L can be partially formed of a silicon film using sapphire as a base material, the LCDs 26 and 27 and the film CCDs 4R and 4L having a toothless shape are integrated into one chip. Can also. However, the transparency (light transmittance) of the chip is on the order of thin sunglasses.

The film CCD 4R captures the pupil movement of the observer's left eye and the aperture state of the pupil shown in FIG. 7B, and the film CCD 4L captures the movement of the pupil of the observer's right eye shown in FIG. , The aperture state of the pupil is imaged.
Although the imaging performance is somewhat lowered, there is no problem in capturing the eye movement of the observer. Therefore, film CCD 4R, 4L
The automatic zoom mechanism of the panning CCD device 23 can be controlled based on the camera output signal S3 obtained from the camera so that the focus is always constant.

The special glasstron 2 is mounted on the face or head of the observer shown in FIG. 8, and the stereo image of the LCD 26 and the stereo image of the LCD 27 are guided to the observer's eyeball. In FIG. 8, the point where the optical axes of both eyes of the observer wearing the special glasstron 2 overlap is the gazing point p. Based on this point of gaze p,
Stereo image of background image to which observer belongs and 3D polygon 1
A stereo image of 0 is synthesized in the head. Here, the separation distance from the surface of the observer's eyeball to the gazing point p is defined as Sh, and the gazing point p is determined from the lens surface of the panning CCD device 23.
Is defined as Sc, and the offset distance between the surface of the eyeball of the observer and the lens surface of the panning CCD device 23 is defined as Soff.

As described above, the eyeball image of the observer is provided on the front surfaces of the left and right LCDs 26 and 27 on the film CCD 4R,
The present invention is not limited to the case where 4L is provided, and outputs a background image from the left-eye display LCD 27 and outputs a right-eye display LCD.
26, a film CCD 4R eye is provided on
An eyeball image may be obtained by using The reverse method may be used. In this example, the positional relationship between the iris and the pupil and the shape of the pupil are recognized from the eye movements of the observer, the direction of the point of gaze p and the separation distance Sh are determined from the recognition information, and the separation distance S is determined.
h based on special glasstron 2 and light emitting diode LE
The focal length Sc + Soff with respect to Di is corrected.

The special glasstron 20 shown in FIG. 9 constitutes a transmissive head-mounted display, and does not include a normal CCD image pickup device 25. Accordingly, the transmissive head-mounted display includes a panning CCD device 23, a liquid crystal shutter 28 for capturing an external image, an LCD 29 as an image display element, and the above-described film CCDs 4R and 4L (not shown). .

For example, a panning CCD device 23 is arranged at a position corresponding to the observer's eyebrow. When the observer turns his / her eyes to the reference surface setting bracelet 1, the four light emitting diodes LED1 of the bracelet 1 are placed. 4 are panned. Liquid crystal shutters 28 are provided at positions corresponding to the left and right eyes of the observer. For example, when the liquid crystal shutter 28 is opened, the real image of the bracelet 1 of the observer passing through the liquid crystal shutter 28 is directly guided to the eyeball. I will In this example, the above-mentioned film CCDs 4R and 4L are located at positions facing the eyeball of the observer and dispersed on the surface of the liquid crystal shutter 28.

An LCD 29 is attached to a portion of the special glasstron 2 located beside the left and right eyes of the observer, and a character image is displayed in the same manner as the special glasstron 2 described above. Although not shown, the liquid crystal shutter 2
Optical means such as a polarization beam splitter is provided between the LCD 8 and the LCD 29 so that the real image of the bracelet 1 of the observer and the image of the 3D polygon 10 are guided to the eyeball of the observer. Thus, the background image to which the observer belongs and the 3D polygon 10 are combined in the head.

Next, the internal structure of the follow-up CCD device 23 of the interline transfer system will be described. FIG.
The panning CCD device 23 shown in FIG. On the substrate 31, photodiodes PHij (i = 1 to n, j =
1 to m) are arranged in a matrix of n columns × m rows.

In the column direction of this substrate, m
Vertical transfer units 32 are provided, and the photodiodes PH
The signal charge read from ij is transferred in the vertical direction (follow-up direction) based on the vertical read signal S1. A horizontal transfer unit 33 is connected to the vertical transfer unit 32, and the signal charges are transferred in the horizontal direction based on the horizontal readout signal S2, so that a follow shot signal SOUT is output to the output terminal. In this example, the signal charge is read out from the photodiode PHij to the vertical transfer unit 32 at least a plurality of times during the same field period in order to perform a panning shot.

The panning CCD device 23 has a fisheye lens 35 shown in FIG. The fisheye lens 35 is provided on the optical axis of the CCD image sensor 36, for example. The fisheye lens 35 allows a wide range of images such as the bracelet 1 for setting the reference plane of the observer.
Of course, a normal lens may be used, but since the field of view is narrowed, the observer must tilt his head more toward the bracelet 1.

In this example, when the blinking pattern of the light emitting diode LEDi is confirmed at the portion closest to the center of the fisheye lens 35, the focus is controlled by the automatic zoom mechanism of the panning CCD device 23 so that the focus is always constant. Fit together. Therefore, it is possible to improve the accuracy of obtaining the blinking pattern of the light emitting diode LED. In this focused state, the light emitting positions of the four light emitting diodes LED1 to LED4 are obtained, and the reference plane is recognized by connecting the positions. At that time, the light-emitting diode LED
If the focal length Sc between i and the panning CCD device 23 is too short to obtain the luminance information of the light emitting diode LED, the focus is returned to normal and the blinking pattern of another light emitting diode LEDi in a wide area is recognized. To do.

Next, the circuit configuration of the virtual image three-dimensional composition device 200 will be described. The virtual image three-dimensional composition device 200 shown in FIG. 12 is roughly composed of three circuit blocks. The first circuit block is a bracelet 1 for setting a reference plane. The bracelet 1 is provided with a blinking control circuit 13, which controls blinking by applying a predetermined voltage to the four light emitting diodes LED1 to LED4. In the blinking control circuit 13, the light emitting diode LED
1, the blinking interval of LED2, LED3 and LED4 is controlled.

In this example, the blink control circuit 1 shown in FIG.
The IC chip 3 is formed into an IC chip having a thickness of about several hundreds to several thousand μm. The IC chip is incorporated in the plate portion 11, and a predetermined voltage is applied to the four light emitting diodes LED1 to LED4 so that the light emitting diodes are controlled to blink. This blinking control circuit 13 has, for example, a clock generator 61. The clock generator 61 includes, for example, a 1/2 frequency divider 62, a 1/3 frequency divider 63, and a 1/4 frequency divider.
A frequency dividing circuit 64 is connected, and a clock signal CLK1 of a predetermined frequency, a clock signal CLK2 obtained by dividing the frequency of the clock signal CLK1 by 1/2 by a 1/2 frequency dividing circuit 62,
Clock signal CL frequency-divided by で in １ ／ frequency dividing circuit 63
K3 and a clock signal CLK4 whose frequency has been divided by a quarter by the quarter frequency dividing circuit 64 are output.

Each of the clock signals CLK1 to CLK4 is supplied to each of the light emitting diodes LED through a stabilizing resistor R.
1, LED2, LED3 and LED4. The clock generator 61 is connected to a DC power supply E via a power switch SW, and a small dry cell or button cell is used as the power supply E.

FIG. 14 shows a light emitting diode LED1, LED
FIG. 2 is a waveform diagram showing an example of voltage supply to LED3 and LED4. In this example, the power switch SW shown in FIG.
Is turned on, the clock signal CLK1 is supplied to the light emitting diode LED1, the clock signal CLK2 is supplied to the light emitting diode LED2, and the light emitting diode LED1 is turned on.
3 is supplied with the clock signal CLK3, and the light emitting diode LED4 is supplied with the clock signal CLK4. Therefore, four light emitting diodes LED1, LED2, LE
Blinking control can be performed so that the blinking patterns of D3 and LED4 are different.

Returning to FIG. 12, in this example, a gyroscope 9 is provided in the bracelet 1, and three-dimensional position information (hereinafter, referred to as a position detection signal S4) based on the posture of the bracelet 1 is sent to the image processing device 3. The output has been made. In the image processing device 3, the position detection signal S
From 4, the focal length Sc between the light emitting diode LEDi and the special glasstron 2 is calculated. This focal length Sc
And the camera output signal S by the film CCDs 4R and 4L
Based on 3 above, the follow-up CCD device 23 can be more precisely focused on the specific light emitting diode LEDi watched by the observer.

The second circuit block is a special glasstron 2
In the case of the non-transmissive type, the panning CCD device 23 described above, the normal CCD image capturing device 25, and the LC for displaying the right eye are used.
D26, LCD 27 for left eye display and left and right films C
CD4R, 4L (hereinafter collectively referred to as a camera 4
).

The third circuit block is the image processing device 3 and has an internal bus 41. An interface (I / O) 42, an image capture unit 43, an image processing unit 44, a CPU 45, a ROM 46, a RAM 47, and an E ² PROM (electrically writable and erasable read-only memory) 48 are connected to the internal bus 41. ing. Panning C
The CD device 23, the normal CCD imaging device 25, the LCD 26 for displaying the right eye, the LCD 27 for displaying the left eye, the camera 4 for detecting the gazing point, and the gyroscope 9
2 and connected to the internal bus 41.

The internal bus 41 has E ² as a recording medium.
The PROM 48 is connected, and the 3D
An algorithm for three-dimensionally synthesizing the image of the polygon 10 is stored. For example, the E ² PROM 48 captures an arbitrarily set reference plane in the real space to which the observer belongs, displays the reference plane in the virtual space, and displays an image of the 3D polygon 10 in the virtual space. An algorithm for superimposing and synthesizing is stored. Therefore, by executing this algorithm, the reference plane in the real space can be easily recognized with a small amount of calculation. As a result, as if the reference plane in the real space belongs to,
Since image processing can be performed so that the 3D polygon 10 exists, a virtual image three-dimensional composition device 200 such as a virtual character breeding game device can be configured with high reproducibility.

Further, a ROM 46 is connected to the internal bus 41, and stores a system program for controlling the virtual image stereoscopic synthesis apparatus 200, control information such as a memory readout procedure, and the like. A working RAM 47 is connected to the internal bus 41, and system information and display information for displaying a character image are temporarily recorded.

For example, the RAM 47 stores position information (x1, y1), (x2, y2), (x3, y) of the four light emitting diodes LED1 to LED4 of the reference surface setting bracelet 1.
3), (x4, y4) and four light emitting diodes LE
Distance information Lx between D1 and LED4 = (x3-x1) =
(X4-x2) or distance information Ly = (y2-y1) =
(Y4-y3) are stored.

The internal bus 41 has C
The PU 45 is connected, and the interface 42, the image capture unit 43, the image processing unit 44, the ROM 46, the RAM 47
And control of input and output of E ² PROM 48 and panning CC
D device 23, CCD imaging device 25, LCD 26, LCD
27, camera 4 and gyroscope 9 for gaze point detection
Is controlled.

For example, the CPU 45 outputs the camera output signal S3
The focal length Sc of the panning CCD device 23 is adjusted on the basis of the movement of the pupil recognized by the observer based on the eye movement of the observer.
It operates to correct the focal length Sc between i and the panning CCD device 23. In another example, the CPU 45 controls the video output to the LCD 26 and the LCD 27 based on the camera output signal S3. For example, the image of the part watched by the observer is enlarged at a predetermined magnification and the LCD 26, the LCD 27
Is displayed.

In this connection, the interface 42 is provided with an operation section 8 as an instruction means, and at least an instruction signal S5 for returning the image of the part watched by the observer to a predetermined size and displaying the image is displayed on the interface 42. Via CP
An instruction is given to U45.

The interface 42 has an image processing unit 4
For example, images of the four light-emitting diodes LED1 to LED4 for setting a reference plane shown in FIG. 15 taken by a normal CCD image pickup device 25 are processed together with a control instruction of the CPU 45 via the interface 42. The image is taken into the unit 44, where predetermined image processing is performed, and the LCD 2 in the special glasstron 2 is again transmitted via the interface 42.
6 and the LCD 27.

In this example, when the observer tilts the reference surface of the bracelet 1 so as to be perpendicular to the imaging surface of the panning CCD device 23, the position detection signal S4 of the gyroscope 9 in the bracelet 1 becomes an image. It is taken into the processing device 3. In the image processing device 3, the position information and the distance information Lx, Ly of the four light emitting diodes LED1 to LED4 of the reference surface setting bracelet 1 read from the RAM 47.
Based on the position detection signal S4 from the gyroscope 9, the actual position information and distance information Lx and Ly of the four light emitting diodes LED1 to LED4 captured by the panning CCD device 23 are calculated with reference to FIG.

The actual four light emitting diodes LED1
Panning CCD so that the position information and the distance information Lx and Ly of the light-emitting diodes LED1 to 4 by the RAM 47 are closer to the position information and the distance information Lx and Ly of the light-emitting diodes LED1 to LED4.
The focusing optics of the device 23 is automatically adjusted. This allows
Four light emitting diodes LE for setting a reference plane shown in FIG.
The corrected images of D1 to D4 can be taken into the image processing unit 44. The corrected image shows four light emitting diodes L
The positions of ED1 to ED4 are the position coordinates (x1, y1), (x2, y2), (x3, y3) of the four bright points p1 to p4,
(X4, y4), and the distance between the four light emitting diodes LED1 to LED4 is Lx = (x3−x1)
= (X4-x2) or distance information Ly = (y2-y1) =
(The image is made to approximate y4-y3.

Returning to FIG. 12, the interface 4
An image capture unit 43 is connected to 2, and receives a control command from the CPU 45 and performs a predetermined capture process of acquiring image data of a blinking pattern input from the panning CCD device 23. The image data of this blinking pattern is expressed as a change in luminance corresponding to the passage of time. The image capture unit 43 includes an image processing unit 4 as an arithmetic unit.
4 is connected, and for the image data on which the predetermined image processing has been performed, the synchronization deviation of the blinking pattern is corrected, or the reference plane to which the observer belongs is obtained.

For example, in the image processing section 44, the panning CC
Panning signal (luminance signal) S output from D device 23
The blinking pattern of OUT is converted into a spatial arrangement pattern that forms an XY plane including four follow-up luminescent spots p1 to p4 in the image area defined by the window W shown in FIG. After that, the arrangement pattern is scanned, and at least the position coordinates (X
1, Y1), (X2, Y2), (X3, Y3), (X
4, Y4) are required. These four bright spots p1 to p4 are used for setting the reference surface of the bracelet 1 attached to the observer.
Three light emitting diodes LED1 to LED4. 4 in real space
The position coordinates of the two light emitting diodes LED1 to LED4 are known, and the position coordinates are (x1, y1), (x2, y2),
(X3, y3) and (x4, y4).

Therefore, the above-mentioned reference plane in the real space can be obtained by calculating a conversion matrix projected onto the mounting positions of the four light emitting diodes LED1 to LED4. Here, the point (xi, yi, 0) on the plane of the real space is moved by a certain translation / rotational motion, and the point projected on the image coordinate system by the perspective transformation is represented by (Xi, Yi). There is a relationship represented by equation (1).

[0075]

(Equation 1)

Where a1 ... a8 are unknown coefficients and C1
External parameters (position and direction) such as CD imaging device 25
And internal parameters such as focal length. These parameters are the position coordinates (x1, y) of a known point in the real space.
1), (x2, y2), (x3, y3), (x4, y
4) and four sets of position coordinates (X1, Y1), (Y2, Y2), (X3, Y3),
If (X4, Y4) exists, it can be obtained by solving the equation (2).

[0077]

(Equation 2)

The position coordinates (X1, Y
1), (X2, Y2), (X3, Y3), (X4, Y
By connecting 4), the reference plane in the real space shown in FIG. 16 is recognized.

More specifically, when the moving image pickup direction is the Y axis on the arrangement pattern shown in FIG. 17 and the direction orthogonal to the Y axis is the X axis, the image processing unit 44 sets the same direction as the moving image pickup direction. Alternatively, the luminance signal values are added in the opposite direction. When this added value is plotted on the X axis, four positions where the luminance signal value plotted on the X axis is the maximum are detected, and X coordinate values X1, X corresponding to the four positions are detected.
2, X3 and X4 are determined. Further, when the acquired image is scanned in the Y direction on the arrangement pattern, among the plurality of luminescent points arranged in the Y direction, the luminescent spot position that first emits light has a Y coordinate value Y1 corresponding to the X coordinate value. , Y2, Y3, Y4
Is required. The distance information between these four bright points is Lx
= (X3-X1) = (X4-X2) and Ly = (Y2-
Y1) = (Y4-Y3).

Here, the position coordinates of the four light emitting diodes LED1 to LED4 in the real space are defined as wi (i = 1 to 4), and the position coordinates wi of the four light emitting diodes LED1 to LED4 on the camera coordinate system. Assuming that the expression vector is Ci, the position coordinates of the four light-emitting diodes LED1 to LED4 on the LCD screen are Pi, the rotation matrix of the panning CCD device 23 is R, and the movement vector is T, the following equation (3) is obtained.
That is, Ci = R · wi + T (3) where Ci = Pi · ki (ki is a scalar). Therefore, the normal CCD imaging device 25
Of the rotation matrix R and its movement vector T,
Using this as a parameter, coordinate conversion between the real space and the virtual space can be easily performed, so that a character image can be synthesized on a reference plane in the virtual space.

Next, the operation of the virtual image three-dimensional image synthesizing apparatus 200 will be described with respect to the virtual image three-dimensional image synthesizing method of the present invention. In this example, in the virtual space, the 3D
In the case where the image of the polygon 10 is synthesized three-dimensionally,
The description will be made on the assumption that the panning of the light emitting diode LEDi and the correction of the focal length are performed in parallel. For example, the observer wears the special glasstron 2 shown in FIG. 5 on the head.

First, in step B1 of the flowchart shown in FIG. 18, in order to set an arbitrary reference plane in the real space to which the observer belongs, the observer places his or her left arm on the left arm, for example, so that the plate portion 11 faces upward. The reference surface setting bracelet 1 is mounted. Thereafter, the blinking control unit 13 is turned on to blink the four light emitting diodes LED1 to LED4 in a predetermined blinking pattern.

Thereafter, the special glasstron 2 is turned on, while the eye movement of the observer watching the light emitting diode LED 3 is imaged by the watching point detection camera 4 in step B 2. Thereafter, based on the pupil movement recognized from the eye movements of the observer in step B3, the observer gazes at the light emitting diode LED3 and the panning CCD device 2
3, the focal length Sc + Soff is corrected.

For example, by calling the step C1 of the subroutine of FIG. 19, the gazing point p of the observer is detected by the gazing point detecting camera 4 such as the missing film CCD 4R or 4L shown in FIG. 7A. Thereafter, in step C2, the separation distance Sh from the surface of the eyeball of the observer to the gazing point p, the light emitting diode LEDi gazed by the observer, and the panning C
The focal length Sc + Soff with the CD device 23 is determined by the CPU.
45 are compared. Here, the separation distance Sh to the gazing point p and the focal length Sc + Sof of the panning CCD device 23.
If the value f coincides, the adjustment of the optical system is not necessary, and the process shifts to step C8 to continue monitoring the eye movement of the observer. If the distance Sh to the gazing point p does not match the focal length Sc + Soff of the panning CCD device 23, the process proceeds to step C3.

In this step C3, the separation distance Sh to the gazing point p is determined by the focal length Sc + of the panning CCD device 23.
It is determined whether it is larger (A) or smaller (B) than Soff. For example, the separation distance Sh to the gazing point p is a panning shot C
If it is longer than the focal length Sc + Soff of the CD device 23, the process proceeds to step C4, where the CPU 45 adjusts the optical system such as the aperture and lens of the follow-up CCD device 23 so as to increase the focal length Sc.

At this time, in the image processing apparatus 3, the RAM 4
Reference surface setting bracelet 1-4 read from 7
The panning CCD device 2 based on the position detection signal S4 from the gyroscope 9 while referring to the position information and the distance information Lx and Ly of the two light emitting diodes LED1 to LED4.
3 actual four light emitting diodes LED1 to
4, the position information and the distance information Lx and Ly are calculated.

The positions and distances Lx and Ly of the four light emitting diodes LED1 to LED4 actually imaged are stored in the RAM.
The focus optical system of the follow-up CCD device 23 is automatically adjusted so that the position information and the distance information Lx and Ly of the light emitting diodes LED1 to LED4 by the image pickup device 47 become even closer. As a result of this adjustment, in step C5, the separation distance Sh and the focal length Sc
If + Soff matches, the process proceeds to step C8, and if not, the process returns to step C4 to continue the adjustment.

On the other hand, if the separation distance Sh from the observer's eyeball surface to the point of gaze p is smaller than the focal length Sc + Soff, the process proceeds to step C6, where the CPU 45
The optical system of the panning CCD device 23 is adjusted so as to reduce the focal length Sc. As a result of the adjustment, if the separation distance Sh and the focal length Sc + Soff match in step C7, the process proceeds to step C8. If not, the process returns to step C6 to continue the adjustment. Thereby, the light emitting diode LED3 and the panning CC that the observer gazes at
The focal length Sc + Soff with the optical system of the D device 23 can be automatically corrected.

The focal length S of the panning CCD device 23
In parallel with the automatic correction process of c + Soff, in step B4 of the main routine in FIG.
6 and the LCD 27 display a stereo image. On the other hand,
A panning shot of a reference plane in real space is performed using the panning CCD device 23. For example, the four light emitting diodes LED1 to LED4 attached to the position where the image of the 3D polygon 10 is to be synthesized are blinked so that the blinking patterns are different, so that the blinking patterns flow in a predetermined imaging direction. It is imaged.

Thereafter, in step B5, image processing is performed to recognize an arbitrarily set reference plane in the real space to which the observer belongs. In the image processing unit 44, for example, a subroutine shown in FIG. 20 is called to execute a video capture process in step D1. Then, the light emitting diodes LED1 to LED4 at the four corners are recognized in step D2. More specifically, the blinking pattern of the luminance signal by the four light emitting diodes LED1 to LED4 captured by the panning CCD device 23 is converted into a spatial arrangement pattern forming an XY plane including four luminescent points p1 to p4. You.

After that, the arrangement pattern is scanned, and
At least the position coordinates (X1,
Y1), (X2, Y2), (X3, Y3), (X4, Y
4) is calculated, and the above-described expressions (1) and (2) are calculated, and the mounting positions of the four light emitting diodes LED1 to LED4 in the real space and the position coordinates (X1, Y) of the four points of the image processing system.
1), (X2, Y2), (X3, Y3), (X4, Y
4) is obtained, and a reference plane is obtained by connecting these four points. Then, in step D3, the image processing unit 44 performs an arithmetic process based on the above equation (3),
The positional relationship between the panning CCD device 23 and the reference plane is detected.

Thereafter, the process returns to step B4 of the main routine in FIG. 18 to superimpose and synthesize the image of the 3D polygon 10 on the reference plane of the virtual space. At this time, in the special glasstron 2 worn by the observer, one of the stereo images obtained by synthesizing the external world image in the real space and the image of the 3D polygon 10 by the LCD 26 is guided to the right eyeball of the observer. The other stereo image obtained by synthesizing the external world image in the real space by the LCD 27 and the image of the 3D polygon 10 is guided to the left eyeball of the observer.

Therefore, although the 3D polygon 10 does not appear on the reference plane in the real space shown in FIG.
In the virtual space shown in FIG.
Can appear. As a result, the background image in the real space to which the observer belongs and the 3D polygon 10 appearing in the virtual space are synthesized in the head, and it is as if the 3D polygon is located at the position to which the reference plane in the real space belongs. 10 can be present.

As described above, according to the virtual image three-dimensional synthesizing apparatus 200 to which the plane recognition apparatus 100 according to the present embodiment is applied, the focus of the panning CCD device 23 on the specific light-emitting diode LEDi watched by the observer is accurately determined. Can be adjusted. Therefore, it is possible to image the four light emitting diodes LED1 to LED4 at the best focus so as to flow in the predetermined imaging direction. At the same time, the luminance signal of the blinking pattern can be optimally image-processed by the image processing device 3, so that the reference surface of the bracelet 1 can be recognized with high accuracy. Moreover, the reference plane in the real space to which the observer belongs can be easily recognized with a small amount of calculation.

Therefore, a virtual character breeding game device or other game device that allows a TV program character or the like to jump out to the position of the reference plane in the real space to which the observer belongs and to play with the observer's palm is constructed. can do. Further, the calculation load on the image processing unit 44 can be reduced as compared with the conventional method, and the cost of these virtual character breeding game devices and the like can be reduced.

(2) Second Embodiment FIG. 22 is a diagram showing an example of a two-dimensional barcode 50 for setting a reference plane used in a virtual image three-dimensional composition device as a second embodiment. In this embodiment, instead of the four light emitting diodes LED1 to LED4, a two-dimensional barcode 50 is provided at the center of a black and white checkered barcode 90 shown in FIG.
Are arranged in advance. Then, the normal CCD image pickup device 25 zooms up the barcode 90, thereby recognizing the two-dimensional barcode 50 at the center thereof, and thereafter, the two-dimensional barcode 50 is used as a reference plane. is there.

In this example, when the observer gazes at the central part of the checkered bar code 90 shown in FIG.
Based on the camera output signal S3 obtained from the camera 4 for detecting the point of gaze, the CPU 45 performs a display control such that the image at the center is enlarged at a predetermined magnification and displayed on the LCD 26 and the LCD 27. Will be When returning the zoomed-up image to its original state, by operating the operation unit 8 described above, an instruction signal S5 is output to the CPU 45, and the image of the central portion watched by the observer is returned to a predetermined size. To be displayed.

The two-dimensional matrix code 50 includes at least an n-row × n-column black-and-white matrix printed in black on a white background shown in FIG. 23, and a black frame portion 51 having the same thickness as the black-and-white matrix. In this example, 5 × 5 pixels surrounded by a black frame portion 51 are the code region portion 52, and of these 25 pixels, 12 pixels shown in FIG. 23 are painted black.

The barcode 50 is provided at a position where an image of the 3D polygon 10 is to be synthesized, for example, on the plate 11 of the bracelet 1 shown in FIG. The two-dimensional matrix code 50 is imaged by the ordinary CCD imaging device 25 instead of the panning CCD device 23. The image processing device 3 described with reference to FIG. 12 is connected to the output stage of the imaging device 25. The image processing device 3 is provided with arithmetic means, and performs image processing on an image pickup signal (luminance signal) output from the CCD image pickup device 25 to obtain a reference plane from the two-dimensional matrix code 50.

For example, the image processing section 44 performs preprocessing. In this process, first, the acquired image is set at 2 with an appropriate threshold.
Valued. Since the barcode portion is printed in black on a white background, the background image and the code area can be separated quite stably by the fixed threshold value. Next, labeling is performed for each connected region of black pixels. The black frame portion 51 of the two-dimensional barcode 50 is included in any of the labeled connection regions. Therefore, considering the size and aspect ratio of the circumscribed rectangle of the connection area, the code area 52
The background image (region) that is unlikely to include the symbol is removed.

Thereafter, a barcode frame is applied to each element of the connected area obtained as a result of the preprocessing. For example, a black area is searched from each side of the circumscribed rectangle toward the inside, and a point sequence of the black frame part 51 is obtained. A line segment is applied to this point sequence by the least squares method. After that, the code area section 52 given to the two-dimensional barcode 50 is recognized.

The above-mentioned reference plane is obtained by calculating a transformation matrix for projecting the four vertices of the black frame 51 to the vertices of a square. Here, the point (xi, yi, 0) on the plane of the real space is moved by a certain translation / rotational motion, and the point projected on the image coordinate system by the perspective transformation is represented by (Xi, Yi). There is a similar relationship between the expressions (1) described in the first embodiment.

Therefore, these parameters are the position coordinates (x1, y1), (x2, y2),
(X3, y3), (x4, y4) and their corresponding four sets of position coordinates (X1, Y1), (Y2,
If (Y2), (X3, Y3) and (X4, Y4) exist, they can be obtained by solving the equation of the above-described equation (2).

The position coordinates (x1, y1) obtained here,
Regarding (x2, y2), (x3, y3), and (x4, y4), assuming that four vertices of a square whose side length is “1”, a plane connecting these four vertices is a reference plane in the real space. Become. Although the black frame portion 51 on the screen is distorted by the attitude of the CCD imaging device 25 and perspective projection, a rectangular vertex on the screen can be projected to each square vertex by an external parameter and an internal parameter. Therefore, since the cubic cube 53 can be created from the position coordinates of the four corners of the two-dimensional barcode 50 in the virtual space shown in FIG. 23, the 3D polygon 10 and the like can be combined with the cubic cube 53.

As described above, according to the three-dimensional virtual image synthesizing apparatus of the present embodiment, the two-dimensional barcode 50 is arranged in advance in the black and white checkerboard code, and the checkerboard code is stored in the normal CCD image pickup device 25. By zooming in with, the two-dimensional barcode 50 can be recognized. As in the first embodiment, the reference plane in the real space to which the observer belongs can be easily recognized with a small amount of calculation. Furthermore, in the virtual space, an image of the 3D polygon 10 can be three-dimensionally synthesized with good reproducibility at a position using the two-dimensional barcode 50 as a reference plane.

Therefore, as in the first embodiment, the position of the reference plane in the real space to which the observer belongs is as if T
It is possible to configure a virtual character training game device or another game device that allows characters of a V program to jump out and play with the palm of the observer.

In each embodiment, the case where the non-transmission type special glasstron 2 or the transmission type special glasstron 20 is used has been described. However, the present invention is not limited to this, and the transmission type and the non-transmission type can be switched. Of course, it is possible to use a special type of glasstron which can be used for both purposes.

In each embodiment, the case where the 3D polygon 10 is simply projected virtually on the reference plane has been described. However, the present invention is not limited to this, and the speaker may be connected to the special glasstron 2 or the special glasstron 20. May be attached, when the 3D polygon 10 jumps out on the reference plane, an onomatopoeic sound may be sounded, or thereafter, the 3D polygon 10 may speak something.

The virtual image three-dimensional synthesizing apparatus 20 of this embodiment
0 is JP-A-10-123453, JP-A-9-304
727, JP-A-9-304730, JP-A-9-21
1374, JP-A-8-160348, JP-A-8-9
4960, JP-A-7-325265, JP-A-7-2
The present invention can be applied to a transmission type head mounted display described in Japanese Patent No. 70714 and JP-A-7-67055.

In this embodiment, the case in which the two-dimensional imaging device of the interline transfer system is used for the panning CCD 23 has been described. However, the present invention is not limited to this, and the case in which the two-dimensional imaging device of the frame transfer system is used. Even if there is, the same effect can be obtained.

In this embodiment, the case where the 3D polygon 10 is displayed on the reference plane has been described. However, the present invention is not limited to this.
The D polygon 10 can also be displayed. For example, the 3D polygon 10 can appear on the three-dimensional coordinate axes x, y, and z with one of the light emitting diodes LEDi constituting the reference plane as the origin O. Therefore, even when the observer takes a gesture to cover a part of the reference plane, for example, the surrounding area overlooking the reference plane (on a circular orbit in the three-dimensional coordinate space or on a random orbit) 3D polygon 10
Can appear.

In this embodiment, the case of the 3D polygon 10 of the snowman has been described with respect to the virtual body image.
It is not limited to this, but pedestal polygons, lights, flames,
Alternatively, it may be an ice polygon or a polygon like armor.

(3) Third Embodiment FIGS. 25 to 27 are image diagrams showing a configuration example of a virtual art museum to which a virtual image three-dimensional composition device as a third embodiment is applied.

In this embodiment, a plurality of light emitting diode blocks BLi (i = 1 to n) are arranged at a predetermined interval on a substantially flat wall surface, and the blocks BLi are shot and shot and recognized by an image processing system. After that, the light-emitting diodes LEDj (j = 1 to 4) in the block are further shot to make the reference plane recognized, and three painting images are displayed in a virtual space including the reference plane. When the observer gazes at one of the painting images, the image is zoomed up and displayed on the special glasstron 2 or the like.

In this example, blocks BL1 to BL8 of eight light-emitting diodes are mounted on a wall of a real space to which the observer belongs, for example, a frontage (width) of about 4 m × height of about 2.5 m.
L8 are arranged at predetermined intervals at the positions shown in FIG. The block BLi of each light emitting diode is
It has a shape like the plate portion 11 of the bracelet 1 shown in FIG. 4, and is provided with four light emitting diodes LED1 to LED4 having different blinking patterns.

Of course, the four light emitting diodes LED1 to LED4 of each block BLi for which a picture image is to be synthesized.
(X1, y1), (x
2, y2), (x3, y3), (x4, y4). Also in this example, four light emitting diodes LED1 to LED4
Is located at each vertex of the rectangle, the distance Lx between the light emitting diodes LED1 and LED3 is (x3-x1), the distance between the light emitting diodes LED2 and LED4 is also Lx, and the distance Lx is (x4-x2) It is.

The light emitting diodes LED1 and LED2
Is (y2-y1), the distance between the light emitting diodes LED2 and LED4 is also Ly, and the distance L
x is (y4-y3). These position information and distance information are registered in advance in the RAM 47 of the image processing system shown in FIG. 12 for use in recognizing the wall surface. In this example, a predetermined data format (not shown) is prepared to identify the light emitting diode block BLi. This data format is divided into two areas. In one area, a block No. identification code assigned to each block BLi is written to identify each block BLi, and in the other area, the light emitting diodes LED1 to LED1 in each block BLi are written.
4, position information and distance information Lx and Ly are written. This block No. identification code, light emitting diode LE
The position information and the distance information of D1 to D4 are collectively called block identification information. Block identification information on eight light emitting diode blocks BL1 to BL8 based on this data format is stored in the RAM 47.

Here, the observer wears the special glasstron shown in FIG. Then, in step E1 of the flowchart shown in FIG. 26, first, the panning CCD device 23 is set to the maximum telephoto state. This maximum telephoto state is when a distant object is viewed in a normal field of view. When an object is viewed from a distance in this manner, the panning CCD device 23 recognizes the block No. identification codes of the eight blocks BL1 to BL8, but the light emitting diodes LED1 to LED4 of each of the blocks BL1 to BL8 blink. Unable to identify patterns. Here, in the image processing system, each of the blocks BL1 to BLB
It is recognized that L8 is flashing synchronously. But R
Block B according to the block number identification code of AM47
The locations of L1 to BL8 can be recognized by the image processing system.

In other words, the process proceeds to step E2, where the block BL1 of eight light-emitting diodes arranged on the wall surface is set.
It is determined by the image processing system whether or not all of BL8 are within the imaging range of the panning CCD device 23. The determination at this time may be performed using pattern recognition or the like. If the eight blocks BL1 to BL8 do not fall within the imaging range, the process proceeds to step E3, and after the focus adjustment mechanism of the panning CCD device 23 is panned (spiral-rotated),
Proceeding to the step E4, the eight blocks BL1 to BL8
Is subjected to recognition processing by the image processing system.

At this time, in the image processing apparatus 3, the RAM 4
Block B of 8 light-emitting diodes read from 7
The actual position coordinates and distance information Lx and Ly of the eight blocks BL1 to BL8 imaged by the panning CCD device 23 are calculated with reference to the block No. identification codes, position information, and distance information of L1 to BL8. . Thereafter, returning to step E2, the image processing system determines again whether or not all the eight blocks BL1 to BL8 arranged on the wall surface fall within the imaging range.

By repeating this process, if all the eight blocks BL1 to BL8 fall within the imaging range of the follow-up CCD device 23 in step E5, the process proceeds to step E5, where each of the blocks BL1 to BL8 is moved. It is detected whether or not the light emitting diodes LED1 to LED4 exist in BL8. At this time, as described in the first embodiment, the block BLi to which the observer pays attention is zoomed up in step E6.

At this time, in the image processing apparatus 3, the position information and the distance information L of the four light emitting diodes LED1 to LED4 are obtained.
x and Ly are the light emitting diodes LED1 to LED1 by the RAM 47.
The focus optical system of the panning CCD device 23 is automatically adjusted so as to be closer to the position information and the distance information Lx and Ly of No. 4. Thereafter, the process returns to step E5. Therefore, when it is recognized in step E5 that four light emitting diodes LED1 to LED4 are present in the block BLi to which the observer pays attention, the process proceeds to step E7, where the image processing system performs surface recognition processing. This processing is as described with reference to FIGS.

Thereafter, the flow shifts to step E8, where it is determined whether or not the wall surface has been recognized. When the wall surface can be recognized, the surface recognition processing is terminated and another data processing is executed. If the wall surface cannot be recognized, the process proceeds to step E9, in which the distance information Lx, Ly of the four bright points is smaller than the distance information (reference value) Lx, Ly read from the RAM 47 shown in FIG. Is detected. RAM
If the calculated distance information Lx, Ly is smaller than the distance information Lx, Ly calculated by 47, the process proceeds to step E10 to zoom up. Thereafter, the process returns to step E7 to perform the surface recognition processing.

In step E9, the distance information Lx, Ly calculated from the distance information Lx, Ly from the RAM 47.
If is larger, the process goes to step E11 and again
It is detected whether the distance information Lx, Ly of the two bright points is larger than the distance information (reference values) Lx, Ly stored in the RAM 47. If the calculated distance information Lx, Ly is larger than the distance information Lx, Ly from the RAM 47, the process proceeds to step E12 to zoom down.

After the zoom-down in step E12, and when the distance information Lx, Ly calculated by the RAM 47 is not larger than the distance information Lx, Ly in step E11, the process returns to step E7 to perform the surface recognition processing. Is performed. Thereafter, the process proceeds to step E8, where it is determined whether the wall surface has been recognized.

By repeating the above process, the specific block BLi watched by the observer can be recognized by the image processing system with good reproducibility. Therefore, when the wall surface is recognized by the image processing system, the surface recognition processing is terminated, and the 3D polygon 10 and the virtual painting can be synthesized in the virtual space using the specific block BLi as the reference plane. .

In this example, the principle of wall surface recognition is applied, and for example, a portrait P1, a ship picture P2, and a mountain picture as shown in FIG. 27 prepared in advance in an E ² PROM or the like shown in FIG. The stereo image of P3 is displayed on the LCD 26 for right eye display and the L for left display in the special glasstron 2.
Displayed on CD27. Of course, the camera 4 for gaze point detection monitors the eye movement of the observer.

When the observer gazes at the portrait P1 on the left side, a camera output signal S3 indicating that "they are gazing at the portrait" is output from the camera 4 for gazing point detection to the CPU 45. Then, a process of enlarging and displaying the portrait P1 is performed using any one of the light emitting diode blocks BL1, BL2, BL3, or BL4 as a reference plane. Therefore, a stereo image in which the portrait P1 is enlarged can be displayed on the LCD 26 for right eye display and the LCD 27 for left display. When returning the zoomed-up portrait P1 to its original state, by operating the operation unit 8 described above, the instruction signal S5 is output to the CPU 45, and the portrait P1 watched by the observer is returned to a predetermined size. The display is made.

As described above, according to the virtual art museum to which the virtual image three-dimensional synthesizing apparatus according to the present embodiment is applied,
In the virtual space, the portrait P1 and the ship painting P shown in FIG.
Since the normal CCD imaging device 25 can focus on one of the three images of the image 2 and the painting P3 of the mountain, the image is zoomed in and displayed. Can be.

Therefore, since the image watched by the observer can be faithfully displayed, the four light emitting diodes LED1 to LED4 are provided on the wall surface of the real space, and the image software is supplied from a CD-ROM or the like. For this reason, a virtual museum or the like can be configured with high reproducibility.

[0131]

As described above, according to the surface recognition apparatus of the present invention, there is provided the control means for adjusting the optical system of the follow-up imaging means based on the output of the imaging means for detecting the gazing point. The focal length between the light source watched by the observer and the imaging means for panning is corrected based on the movement of the pupil recognized from the eye movement of the observer.

With this configuration, it is possible to focus the panning imaging means on the specific light source watched by the observer, so that it is possible to perform imaging with the light source flowing in a predetermined imaging direction at the best focus. In addition, since the luminance signal of the blinking pattern can be optimally image-processed, the reference plane can be accurately recognized.

According to the surface recognition method of the present invention, the focal length between the light source watched by the observer and the first image pickup system is corrected based on the movement of the pupil recognized from the eye movement of the observer. The luminance signal of the blinking pattern captured by the first imaging system whose focal length has been corrected is subjected to image processing to obtain three positions of the light source, and thereafter, the reference surface is connected by connecting the three light source positions. Is what you want.

With this configuration, it is possible to focus the panning imaging means on a specific light source watched by the observer, so that it is possible to perform imaging with the light source flowing in a predetermined imaging direction at the best focus. In addition, since the luminance signal of the blinking pattern can be optimally image-processed, the reference plane can be accurately recognized. Therefore, this surface recognition method can be sufficiently applied to the surface recognition means of the virtual image three-dimensional composition device.

According to the virtual image three-dimensional composition device of the present invention,
Since the above-described surface recognition apparatus is applied, it is possible to focus the panning imaging unit on a specific light source watched by the observer. Therefore, the light source can be imaged so as to flow in the predetermined imaging direction at the best focus, and the luminance signal of the blinking pattern can be optimally image-processed, so that the reference plane can be recognized with high accuracy.

The present invention is extremely suitable when applied to a three-dimensional display device or the like that synthesizes an image of a virtual body on a reference plane in a virtual space.

[Brief description of the drawings]

FIG. 1 is a plane recognition apparatus 10 according to an embodiment of the present invention.
FIG. 4 is a block diagram showing an example of the configuration of 0.

FIG. 2 is a flowchart showing an operation example of the surface recognition device 100.

FIG. 3 is a perspective view illustrating a configuration example of a virtual image three-dimensional composition device 200 to which the surface recognition device 100 according to the first embodiment is applied.

FIG. 4 is a perspective view showing a configuration example of a reference surface setting bracelet 1 used in the first embodiment.

FIG. 5 is a conceptual diagram viewed from the front showing a configuration example of a special glasstron 2 used in each embodiment.

FIG. 6 is a conceptual view showing an example of the internal configuration of the special glasstron 2 as viewed from the top of a partially crushed state.

FIG. 7 is a conceptual diagram showing a configuration example of the film CCDs 4R and 4L as viewed from above.

FIG. 8 is a conceptual diagram showing an example of a positional relationship of a gazing point p when the special glasstron 2 is mounted.

FIG. 9 shows another special glasstron 2 used in each embodiment.
It is the conceptual diagram seen from the front which shows the example of a 0 structure.

FIG. 10 is a plan view showing an example of the internal configuration of the panning CCD device 23 of the special glasstron 2.

FIG. 11 is a sectional view showing a configuration example of an optical system of the panning CCD device 23.

FIG. 12 is a diagram illustrating an example of a circuit block of the virtual image three-dimensional composition device 200.

FIG. 13 is a block diagram showing an example of the internal configuration of the blink control circuit 13.

FIG. 14 is a waveform diagram showing a voltage supply example of the four light emitting diodes LED1 to LED4.

FIG. 15 is an image diagram showing an example of a normal image of the plate section 11 forming the reference plane.

FIG. 16 is an image diagram showing an example of correction of an image by a gyroscope.

FIG. 17 is a schematic diagram showing an example of calculating the position coordinates of the reference plane.

FIG. 18 is a flowchart of a main routine illustrating an operation example (part 1) of the virtual image three-dimensional composition device 200.

FIG. 19 is a flowchart of a subroutine showing an operation example (part 2) of the virtual image three-dimensional composition device 200.

FIG. 20 is a flowchart of a subroutine showing an operation example (part 3) of the virtual image three-dimensional composition device 200.

21A is a real image of a plate unit 11 in a real space, and FIG. 21B is an image diagram showing an example of a combination of 3D polygons on a reference plane in a virtual space.

FIG. 22 is a plan view showing an example of the arrangement of two-dimensional barcodes 50 in a checkered barcode 90 used in the second embodiment.

FIG. 23 is a plan view showing a configuration example of the two-dimensional barcode 50.

FIG. 24 is a perspective view showing a synthesis example of the 3D polygon 10 on the two-dimensional barcode 50.

FIG. 25 is an image diagram showing an arrangement example of light emitting diode blocks BL1 to BL8 according to the surface recognition method as the third embodiment.

FIG. 26 is a flowchart showing an example of recognition of the wall surface.

FIG. 27 is an image diagram showing a configuration example of a virtual museum applying the surface recognition principle.

[Explanation of symbols]

1 ... Bracelet (object) for reference plane setting, 2, 2
0 ... Special glasstron, 3 ... Image processing device, 4
... Camera for gazing point detection (imaging means for gazing point detection), 5 ... Control means, 6 ... Calculating means, 8 ...
Operation unit (instruction means), 9 ... gyroscope, 10
... 3D polygon, 11 ... plate part, 12 ...
Bracelet part, 13 blinking control circuit, 23 panning CCD device (imaging means for panning), 24 display means, 25 CCD imaging device, 26 right eye LCD for display, 27 ... LCD for left eye display, 32
..Vertical transfer section (charge transfer section), 33... Horizontal transfer section, LEDs 1 to 4... Light emitting diode (light source), 50
... 2D barcode, 90 ... Barcode, 10
0: surface recognition device, 200: virtual image stereoscopic synthesis device

 ────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Tsukasa Yoshimura 6-35 Kita Shinagawa, Shinagawa-ku, Tokyo Sony Corporation F-term (reference) 5B050 BA09 DA01 EA07 EA19 EA28 FA02 FA06 FA08 5B087 AA07 AE00 BC05 BC12 BC13 BC16 BC26 BC32 DD03 DJ00

Claims (12)

[Claims] 1. An object mounted at a desired position on an arbitrary object.
At least three or more light sources that blink so that the blinking patterns are different, an image pickup unit for taking a picture so that the light source is caused to flow in a predetermined imaging direction, and image processing of a luminance signal of the blinking pattern by the image pickup unit Calculating means for determining the position of three points of the light source and then connecting the positions of the three light sources to obtain a reference plane; and capturing the eye movement of an observer who gazes at the reference plane to obtain the observer An image pickup unit for detecting a point of interest for detecting the point of interest, and a control unit for adjusting an optical system of the image pickup unit for following shot based on an output of the image pickup unit for detecting the point of interest. Is characterized in that the focal length between the light source and the imaging means for panning is corrected based on the movement of the pupil recognized from the eye movements of the observer, Surface recognition device. 2. The gyroscope according to claim 1, wherein an object to which the light source is attached is provided, and a gyroscope that outputs three-dimensional position information based on the posture of the object is provided. Surface recognition device. 3. A first imaging system in which three or more light sources are attached to desired positions of an arbitrary object, at least the light sources are blinked so as to have different blinking patterns, and the light sources are caused to flow in a predetermined imaging direction. A second imaging system captures the eye movement of the observer who gazes at the light source with the second light source, and the light source gazes at the observer based on the movement of the pupil recognized from the eye movement of the observer. Correcting the focal length between the light source and the first imaging system; and performing image processing on the luminance signal of the blinking pattern imaged by the first imaging system with the corrected focal length, to obtain three positions of the light source. And a step of connecting the positions of the three light sources to obtain a reference plane. 4. An apparatus for stereoscopically combining an image of a virtual body with an external image to which an observer belongs, a plane recognition means for recognizing an arbitrary reference plane in a real space to which the observer belongs, and the plane Synthesizing means for synthesizing the image of the virtual body on the reference plane of the virtual space recognized by the recognizing means, wherein the surface recognizing means is attached to a desired position of an arbitrary object, and at least a blinking pattern is different. Three or more light sources that blink on and off; an imaging means for panning to image the light source so as to flow in a predetermined imaging direction; Calculating means for obtaining the reference plane by connecting the positions of the three light sources, and detecting the gaze point of the observer by imaging the eye movements of the observer watching the reference plane. Imaging means for viewpoint detection; Control means for adjusting the optical system of the imaging means for panning based on the output of the imaging means for note point detection, the control means of the pupil recognized from the eye movement of the observer A virtual image three-dimensional composition apparatus, wherein a focal length between the light source watched by the observer and a panning imaging unit is corrected based on a movement. 5. The image processing apparatus according to claim 1, wherein said synthesizing unit and the image capturing unit for detecting a point of interest are provided, and a control unit for controlling a video output to the combining unit based on an output of the image capturing unit for detecting the point of interest is provided. 5. The virtual image stereoscopic device according to claim 4, wherein the control unit controls the synthesizing unit so that at least an image of a part watched by the observer is enlarged and displayed at a predetermined magnification. 6. Synthesizer. 6. A case in which the combining means is provided, and at least an instruction means for giving an instruction to the combining means to return an image of a part watched by the observer to a predetermined size and display the image is provided. 5. The virtual image three-dimensional composition device according to claim 4, wherein: 7. A two-dimensional imaging device having a plurality of photoelectric conversion elements constituting each pixel is used as the panning imaging means, and the signal charges obtained from the photoelectric conversion elements are transferred in a predetermined direction. 5. The virtual image three-dimensional composition device according to claim 4, wherein the signal charge is read from the photoelectric conversion element at least a plurality of times during the same field period. 8. The method according to claim 1, wherein the imaging unit and the calculation unit are provided, wherein the calculation unit determines a blinking pattern of a luminance signal by the imaging unit.
By converting into a spatial arrangement pattern forming an XY plane including three luminescent points, scanning the arrangement pattern to obtain at least the position coordinates of three luminescent points, and connecting the position coordinates of the three points 5. The virtual image three-dimensional composition apparatus according to claim 4, wherein the reference plane is recognized. 9. The panning imaging, wherein the panning imaging direction is set to a Y axis on an arrangement pattern forming an XY plane including three bright points, and an axis orthogonal to the Y axis is set to an X axis. The luminance signal values are added in the directions and plotted on the X-axis. The position where the luminance signal value plotted on the X-axis is maximum is detected to obtain three X-coordinate values. When scanning in the Y-axis direction in (1), among the plurality of bright spots arranged in the panning imaging direction, the position of the bright spot that first emits light is obtained as a Y coordinate value corresponding to the X coordinate value. Item 10. A virtual image three-dimensional composition device according to Item 8. 10. The surface recognition means is attached to a position where an image of a virtual body is to be synthesized.
At least an n-row x n-column black-and-white matrix printed in black on a white background, a two-dimensional matrix code including a black frame having the same thickness as the black-and-white matrix, imaging means for imaging the two-dimensional matrix code, and imaging 5. The virtual image three-dimensional composition apparatus according to claim 4, further comprising calculation means for processing a luminance signal by the means to obtain a reference plane from the two-dimensional matrix code. 11. A case in which the synthesizing unit and the imaging unit for detecting a gazing point are provided, wherein the synthesizing unit includes: an imaging unit that captures an external image to which an observer belongs; A first image display element for displaying one of a stereo image synthesized with the prepared image of the virtual body; and a second image display element for displaying the other of the stereo images. 5. The virtual image stereoscopic device according to claim 4, wherein the image pickup means is arranged at a position facing the eyeball of the observer and dispersed in display surfaces of the first and second image display elements. Synthesizer. 12. The liquid crystal shutter according to claim 12, wherein the combining means and the imaging means for detecting a gazing point are provided, wherein the combining means opens and closes incident light to capture an external image to which an observer belongs; An image display element for displaying an image of the virtual object for synthesis with the external image, an image of the virtual object by the image display element, and an external image to which the observer who has passed the liquid crystal shutter belongs to the observer's eyeball. 5. An optical device for guiding, wherein the imaging device for detecting the gazing point is disposed at a position facing the eyeball of the observer and dispersed on the liquid crystal shutter surface. The three-dimensional virtual image synthesizing apparatus according to claim 1.