JP2000182058A - Three-dimensional motion input method and three- dimensional motion input system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To efficiently input motion information by spatially and timewise automating a work to specify a corresponding point of a motion extracting point when three-dimensional motion information is inputted by using a stereo photographing system. SOLUTION: A three-dimensional motion input method is constituted by providing a photographing process to photograph a pair of moving images with parallax, display processes (S8002, S8014) to perform stereo display of the photographed pair of moving images to be streoscopically viewed, a line of sight detecting process (8010) to detect the direction of the line of sight of an observer to observe the image to which the stereo display is performed, a corresponding point detecting process (S8011) to detect the corresponding point of the pair of moving images based on the detected direction of the line of sight, a calculating process (S8011) to calculate a three-dimensional coordinate of a point on the moving image corresponding to the detected corresponding point and a storage process (S8012) to store the three-dimensional coordinate by making it correspond to time.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a three-dimensional motion input method and a three-dimensional motion input system, and more particularly, to a three-dimensional motion input method and a three-dimensional motion input method for inputting a three-dimensional motion of an actual object by a stereo photography method. The present invention relates to a three-dimensional motion input system.

[0002]

2. Description of the Related Art In recent years, digital video synthesizing techniques using computer graphics (hereinafter abbreviated as CG) have been frequently used for producing special effects scenes such as movies. This is used to create a scene that is impossible or difficult to shoot with a real shooting method, and replaces a specific object in a real shooting scene with an artificial image created by CG, or replaces a new CG image with a new CG image. It is a method of adding. In this case, in order to more accurately combine the real image and the CG image, three-dimensional coordinate information of the position of the object to be replaced in the real image and the CG image additional position and its time change, that is, three-dimensional motion information Need to be known in some way.

On the other hand, the above-mentioned digital image synthesizing technique is considered to complement an image of a real object with an artificial image of a CG. There is also a great need for technology to provide characteristics. In other words, this is a technique in which three-dimensional motion information of a real object is extracted as a numerical value by some method, and this motion information is reflected in an image generation process by CG. The motion characteristic extraction processing performed in this application is called motion capturing. An example of the use of motion capturing is in a video or the like in which a character created by computer graphics dances like a real dancer. Also, in recent computer games and the like, it is generally used to make the movement of a character realistic.

Against this background, in recent years, a technique of extracting three-dimensional motion information from a real moving object by photographing or measuring, that is, a technique of three-dimensional motion input, has been increasingly required in recent years.

[0005] Three-dimensional motion input is performed by focusing on a specific point of a moving object, and extracting three-dimensional information of this point in chronological order. Currently available methods for realizing this three-dimensional motion input include a magnetic sensor system, an optical system, and a stereo shooting system. Hereinafter, the principles of these methods will be briefly described.

[0006] 1. Magnetic sensor system The magnetic sensor system is currently widely used as a powerful means for inputting accurate three-dimensional motion. In this method, a magnetic sensor is mounted on a measurement target, and a signal from a signal source called a transmitter is detected by the magnetic sensor to acquire a spatial position and a direction of the magnetic sensor. Since this magnetic sensor system can directly obtain three-dimensional coordinates at high speed, it is possible to input three-dimensional coordinates in real time. In addition, since information on both the position and the direction can be obtained for each sensor, highly accurate three-dimensional motion input is possible. For this reason, the magnetic sensor system is currently widely used for motion capturing. However, in the magnetic sensor method, it is necessary to mount a sensor, which imposes a certain load on the movement of the measurement target and at the same time limits the movable area. Further, there is a restriction that the measurement is limited to a photographing studio in which the measurement device is provided.

[0007] 2. Optical method The optical method is a method in which an optical marker is attached to a measurement target and three-dimensional motion is extracted by photographing a plurality of cameras whose positions and directions are known in advance. In this case, a target to be measured is irradiated with a specific wavelength such as infrared light at the time of photographing, and a material that strongly reflects the specific wavelength is used as a marker. This makes it possible to shoot a video image consisting of only the marker image, and to calculate the coordinate information of the marker from the captured image of the marker and the coordinate information of each camera. Although this method does not apply as much load to the movement of the measurement target as the magnetic sensor method, the measurement is limited to a special photographing studio in which a measuring device is arranged, similarly to the magnetic sensor method.

[0008] 3. Stereo Photography Method As a method for extracting three-dimensional information of an object from a plurality of still images, a method using stereo camera photography is known.
This is because, as shown in FIG.
Two images with parallax taken from two viewpoints by two or two shootings are used as information sources. By using the left and right two images 103 and 104 having the parallax and designating corresponding points in the left and right two images of the point of interest, three-dimensional coordinates of the point of interest can be obtained based on the principle of triangulation. Or later,
This method of extracting three-dimensional coordinates by stereo camera imaging and corresponding point designation will be referred to as a stereo imaging method. If this stereo shooting method is applied to a series of time-series images, a three-dimensional motion input is possible in principle.

As described above, for the purpose of three-dimensional motion input, only motion information of a three-dimensional motion information source is required, such as motion capturing, and video information may not be necessary. In this case, a magnetic sensor system or an optical system is effective and is often used in practice. However, there is a disadvantage that the photographing target is limited. On the other hand, there is also a need to perform three-dimensional motion input and imaging from the same event information source. That is, for example, when it is desired to obtain three-dimensional motion information of a specific photographed object from an image obtained by actual photographing. In this case, in the magnetic sensor system or the optical system, the image of the sensor or the marker is mixed in the captured image, so that the captured image itself can hardly be used. Another limitation is that it is limited to shooting in a special shooting studio. Therefore, in order to perform three-dimensional motion input and imaging in the same event information source, the stereo imaging method is the most suitable choice.

[0010] As described in the prior art, there is a great need for three-dimensional motion input in a situation where the conventional magnetic sensor system or optical system cannot be applied. At present, the stereo photography method is considered to be the most effective method to satisfy the demand.

Further, the stereo photographing method can be realized at a lower cost than the magnetic sensor method and the optical method. In addition, a device configuration in which only one camera is added to the normal shooting for image acquisition is sufficient, and the novelty for shooting for image acquisition is high. Therefore, there is a great advantage in using the magnetic sensor system or the optical system instead. However, the following disadvantages existed in the three-dimensional motion input by the stereo photography method.

First, in deriving three-dimensional coordinate information from the left and right images, corresponding points in the two left and right images have to be specified. This requires an operation of manually specifying each corresponding point on the two left and right images with respect to a specific point of interest using a mouse or the like. This operation focuses on the left (or right) image 103 as shown in FIG.
This is a simple operation of selecting No. 5, then paying attention to the right (or left) image 104, and selecting a point 106 corresponding to the previously selected representative point with the mouse. However, when performing a three-dimensional motion input, it is necessary to perform this operation at all times, which not only places a heavy burden on the operator but also takes time.

As a method for reducing the task of designating corresponding points, there is known a method of calculating correlation between left and right images and automatically calculating corresponding points. This defines the degree of correlation for two points in the left and right images, and for a point given on the left (or right) image, the point on the right (or left) image that maximizes the degree of correlation is the corresponding point It is assumed that. The calculation of the degree of correlation is performed by defining, on the left and right images, rectangular areas of equal size surrounding the two points of the left and right images, respectively, and within this rectangular area, left pixel value data L (x, y ) And the left pixel value data R (x, y). However, the automatic detection of the corresponding points by the correlation calculation also has several problems, such as a point that a lot of calculation time is required, a point that it is necessary to specify a motion extraction point in at least one of the left and right images, and the like. Yes,
At present, no significant improvement in the corresponding point detection processing has been obtained. Therefore, when using the conventional stereo photography method, three-dimensional motion input is performed for a long time.

The operation of specifying the corresponding points of the left and right images is a correspondence between two coordinate systems based on the two left and right viewpoints, and can be said to be a specification of a spatial corresponding point. On the other hand, when the stereo shooting method is applied to a time-series image, it is necessary to specify temporally corresponding points in addition to the spatially corresponding points. That is, it is necessary to specify which point on the image at the next time corresponds to one point of the object in the three-dimensional space specified by one point on the image at a certain time. The following problem arises with regard to the designation of the corresponding points in time.

Here, let us consider a case where a time series image of either the right image or the left image is continuously displayed on the same screen. In this case, it is easy to focus on a specific point of the photographed object and track that point over time. This is because it is easy for humans to recognize moving images easily by paying attention to the extraction points that change continuously on the screen. However, when the corresponding points are displayed sequentially at different positions on the right and left time-series screens and the corresponding points are sequentially specified, the corresponding points of the left and right images must be specified for each time. FIG.
Represents this situation. Therefore, naturally, the temporal continuity of the image is interrupted, and the line of sight of the operator moves back and forth between the left and right images located at different positions (11001, 11003).
For these reasons, in the method of sequentially applying the conventional stereo shooting method to the time-series stereo image, it is easy to lose the tracking between the frames of the point of interest, that is, to lose track of which point was focused as the motion extraction point It is easy, and in some cases, it is necessary to compare images between temporally preceding and succeeding frames (11002).

[0016]

As described above,
In the three-dimensional motion input by the conventional stereo shooting method, the point where it takes time to specify the left and right image corresponding points at the time of extracting the three-dimensional coordinates of one point, and the tracking of the motion extraction point on the time axis,
That is, there is a problem in that it is difficult to specify a corresponding point in time.

The present invention has been made in view of the above problems, and when three-dimensional motion information is input using a stereo shooting method, the operation of designating a corresponding point of a motion extraction point is performed spatially and temporally. It is also an object of the present invention to provide a three-dimensional motion input method and a three-dimensional motion input system that can automate motion information and efficiently input motion information.

[0018]

In order to achieve the above object, a three-dimensional motion input method according to the present invention comprises: a photographing step of photographing a pair of moving images having parallax; The moving image is displayed in a stereoscopic manner so as to be stereoscopically viewed, a line-of-sight detection step of detecting a line-of-sight direction of an observer who observes the image stereo-displayed in the display step, and is detected in the line-of-sight detection step. A corresponding point detecting step of detecting a corresponding point of the pair of moving images based on the line of sight direction, and calculating three-dimensional coordinates of a point on the moving image corresponding to the corresponding point detected by the corresponding point detecting step. A calculating step; and a storing step of storing the three-dimensional coordinates obtained in the calculating step in association with time.

Alternatively, a reading step of reading a pair of moving images having parallax from a recording medium, and a display for stereoscopically displaying the pair of moving images read from the recording medium in the reading step so as to be stereoscopically viewed. Step, a line-of-sight detection step of detecting a line-of-sight direction of an observer who observes the image stereo-displayed in the display step, and a corresponding point of the pair of moving images based on the line-of-sight direction detected in the line-of-sight detection step. A corresponding point detection step of detecting the three-dimensional coordinates of a point on a moving image corresponding to the corresponding point detected in the corresponding point detection step, and the three-dimensional coordinates obtained in the calculation step And storing the information in association with time.

Further, the three-dimensional motion input system according to the present invention includes a photographing means for photographing a pair of moving images having parallax, and a stereoscopic image for stereoscopically viewing the pair of moving images photographed by the photographing means. Display means for displaying, gaze detection means for detecting a gaze direction of an observer observing an image stereo-displayed on the display means, and the pair of moving images based on the gaze direction detected by the gaze detection means Corresponding point detecting means for detecting a corresponding point of, a calculating means for calculating three-dimensional coordinates of a point on a moving image corresponding to the corresponding point detected by the corresponding point detecting means, and a tertiary coordinate obtained by the calculating means. Storage means for storing the original coordinates in association with the time.

Alternatively, reading means for reading a pair of moving images having parallax from a recording medium, and reading the pair of moving images read from the recording medium by the reading means,
Display means for performing stereo display so as to be stereoscopically viewed, gaze detection means for detecting a gaze direction of an observer observing an image stereo-displayed on the display means, and a gaze direction detected by the gaze detection means Corresponding point detecting means for detecting corresponding points of the pair of moving images, calculating means for calculating three-dimensional coordinates of points on the moving image corresponding to the corresponding points detected by the corresponding point detecting means, Storage means for storing the three-dimensional coordinates obtained by the calculation means in association with time.

According to the above configuration, when extracting three-dimensional motion information from a real image by a stereo shooting method, the amount of work and the time required for extracting a time series of three-dimensional coordinates can be greatly reduced. It is easy to use a stereo imaging method having a merit of eliminating the need for a sensor as a three-dimensional motion input method, and it is possible to broaden an imaging target of the three-dimensional motion input.

According to a preferred aspect of the present invention, in the photographing step, the photographing means photographs a moving image with the photographing position and direction fixed.

According to another preferred aspect of the present invention, in the photographing step, a moving image is photographed while moving the photographing position of the photographing means on a predetermined trajectory.

With the above configuration, the present invention can be used for inputting the shape of a stationary object, and the accuracy of shape display can be easily improved.

Further, according to a preferred aspect of the present invention, in the corresponding point detecting step, the corresponding point detecting means detects a point on a stationary object in the moving image and a point on a moving object in the moving image. Each corresponding point is detected, and in the calculating step, three-dimensional coordinates of a point on the moving object with reference to the point on the stationary object are calculated. Thus, even when the photographing position changes arbitrarily, three-dimensional coordinates based on a stationary object in the photographed image can be obtained, so that the degree of freedom in photographing is improved.

Further, according to a preferred aspect of the present invention, in the displaying step, a pair of images at one time of the pair of moving images is displayed on the display means at predetermined time intervals, respectively. The image is updated and displayed, and in the calculation step,
The calculating means calculates three-dimensional coordinates of a point on a moving image at the predetermined time interval.

According to another preferred aspect of the present invention, in the display step, a pair of images at one time of the pair of moving images is displayed on the display means based on an external instruction. Update and display the image at the next time,
In the calculating step, three-dimensional coordinates of a point on the pair of images updated by the calculating unit in the image updating step are calculated. This allows the observer to freely instruct the timing of the motion input, thereby reducing the burden on the observer.

According to a preferred aspect of the present invention, in the calculating step, a two-dimensional area near the corresponding point obtained in the corresponding point detecting step in the pair of moving images by the calculating means is used. A dimensional correlation operation is performed, a new corresponding point on the pair of moving images is obtained based on the obtained correlation value, and three-dimensional coordinates of a point on the moving image are calculated based on the newly obtained corresponding point. .

According to a preferred aspect of the present invention, in the line-of-sight detecting step, the non-visible light source in the line-of-sight detecting means irradiates the observer's eye with invisible light, and the line-of-sight detecting means The rotation angle of the observer's eyeball is obtained from the relative relationship between the observer's eyeball pupil center position and the virtual image position of the invisible light from the corneal reflection surface, and the gaze direction is detected.

According to a preferred aspect of the present invention, the line-of-sight detecting means has a light source for irradiating the observer's eye with invisible light. The rotation angle of the observer's eyeball is obtained from the relative relationship between the virtual image positions of the invisible light and the gaze direction is detected.

Preferably, the display means is a head mounted stereoscopic image display device capable of fixing a relative positional relationship between a viewer's head and a display surface.

Further, more preferably, the head mounted image display device has the line of sight detecting means.

According to a preferred aspect of the present invention, the display means is a three-dimensional image display device having a mechanism for detecting a position of an observer's head relative to the display means. A relative position change of the observer's head with respect to the display device is detected, and a line-of-sight direction in the line-of-sight detecting means is corrected based on the detected relative position change.

[0035]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will be described below in detail with reference to the accompanying drawings.

<First Embodiment> A three-dimensional motion input system according to the present embodiment is composed of a stereo image photographing device, a head mounted stereoscopic image display device, a visual line detection device, and a computer.

FIG. 2 is a conceptual diagram of the overall configuration of the three-dimensional motion input system according to the present embodiment. The stereo image capturing devices 201 and 202 capture a moving object as a time-series stereo image. The moving object is 20 at time 1
3a, at time 2 at 203b. The photographed time-series stereo images are input to the computer 204,
Stereoscopic image display device 20 with head mounted gaze detection device
5 is displayed. At this time, the image of the moving object 203a at time 1 and then the image of the moving object 203b at time 2 are displayed in this order. During the work of the motion input, the observer 200 displays the virtual object 2 displayed on the stereoscopic image display device 205 in chronological order.
The user continues to gaze at one of the points 06a and 206b. At this time, the gaze of the observer 200 is detected by the gaze detection device 205.
The gaze information is input to a computer, and the computer analyzes the gaze information in chronological order to derive three-dimensional motion information 207.

(Stereo shooting) 201 and 2 in FIG.
Reference numeral 02 denotes a stereo image capturing device (hereinafter, referred to as a “left-eye camera” and a “right-eye camera”, respectively) used in the embodiment of the present invention. In this embodiment, two digital video cameras are used. Left eye camera 201
The right-eye camera 202 and the right-eye camera 202 are separated from each other by a predetermined distance, and are arranged so that their optical axes are parallel to each other. In this embodiment, a digital video camera is used to obtain a time-series image, but an analog video camera may be used.

(Recording of Time Code at the Time of Shooting) The images shot by the two left and right video cameras 201 and 202 are respectively recorded on separate recording media such as recording tapes. When reproducing the recording tape, that is, when generating a stereoscopic image, it is necessary to synchronize the time of the left and right images. That is, if the image at the time of interest “time T” is selected from the video captured by the right-eye camera 202, the left-eye camera 2
An image at the same time as “time T” must be selected from the image 01 and a set of stereo images must be created. of course,
If the recording tape is completely rewound at the start of shooting, the two left and right cameras start shooting simultaneously, and the tape is completely rewound during playback of the recording tape, the time synchronization can be obtained from the tape travel distance. . However, since such a method is actually inconvenient, the video camera preferably has a time code recording function on a recording tape. Although most video cameras have a time code recording function, even if the camera does not have a time code recording function, the time of the left and right images can be specified immediately before and after the phenomenon to be captured. May be intentionally inserted to substitute for the time code.

Although two video cameras are used in the present embodiment, the optical system of one video camera is devised without using two video cameras, and an optical image detecting device, for example, an image forming surface of a CCD is used. The images from the left and right two viewpoints may be displayed on the top half each. In this way, it is possible to use only one video camera.

Further, the photographing device used is not limited to an analog video camera or a digital video camera used for ordinary moving image photographing, and it is possible to photograph and record time-varying phenomena on an arbitrary recording medium at sufficiently short time intervals. Obviously, any possible photographing device is applicable to the present invention.

FIG. 3 shows the configuration of the computer 204 used for implementing the present invention. Computer 20
The main storage device 3003 stores a program for deriving three-dimensional motion information from the line-of-sight information, and this program is executed by the CPU 3004 of the computer. The time-series stereo image recorded on the recording medium 3011 is input via the recording medium reproducing device 3012 and the video signal input circuit 3002, and is stored in the secondary storage device 3015 as a time-series stereo image of digital information.
The stereo image to be subjected to the three-dimensional coordinate extraction processing is stored in the image memory 3005, and is displayed on the three-dimensional image display device 3013.
It is output through the video signal output circuit 3006. The gaze data sent from the gaze detection device 3014 is received by the gaze data input circuit 3007,
4 is taken in. Input devices such as a mouse 3008 and a keyboard 3009 are used for the interaction between the operator and the computer.

Before describing the three-dimensional image display device 3013 and the visual line detection device 3014, first, the principle of stereoscopic vision and the principle of deriving three-dimensional coordinate information of a point of interest and further three-dimensional motion information from visual line information. Will be described briefly.

(Fusing operation) The principle that a human can obtain a stereoscopic effect by a stereoscopic image display device using a set of stereo still images is well known. Here, this principle will be briefly described. deep.

4 is an object 403 and an object 403.
It is a view of the state of the eyeball of the observer who observes from above,
FIG. 4B shows the state of the observer's eyeball and the stereoscopic image display device 3.
FIG. 13 is a view of the image No. 013 viewed from above. When the human gazes at the point of interest 402 of the object 403 in the three-dimensional space, FIG.
As shown in (a), the left and right optical axes of the eyeball are the point of interest 402.
Adjust to cross. This operation corresponds to matching the positions of the left and right images on the retina of the left and right eyeballs, that is, the positions of the points of interest 402, and is called “fusion”. A stereoscopic image display device using stereoscopic vision provides a stereoscopic effect by artificially inducing this fusion operation to an observer. That is, as shown in FIG. 4B, the left and right images 405 and 406 having a parallax are devised by some method so as to be separately input to the left and right eyes. At this time, the observer watching the display device controls the optical axis of the eyeball as shown in FIG. 4B in order to fuse the left and right images of the point of interest with parallax. As a result, it is illusioned that a point of interest of the object exists at a point where the optical axes intersect, that is, at a point 407 in FIG.

In FIG. 4B, the left and right images 405 and 406 are displayed at positions shifted forward and backward for convenience of explanation, but are actually displayed at the same position. In the present embodiment, as shown in FIG.
The left and right images displayed on the left and right are displayed on different optical systems 5.
Observe through 03,504. For this reason, it is devised that virtual images of different LCD images displayed at the same display position 505 are separately input to the left and right eyes.

The fusion operation described above is considered to be performed completely unconsciously by natural pattern matching of the left and right images performed by the human brain. In the present invention, the corresponding points of the left and right images are derived using the natural fusion operation of a human. That is, the line of sight of the observer who has fused the point of interest points to the corresponding point of the point of interest on the left and right images. Therefore, by detecting the line of sight of the observer,
It is possible to know the corresponding point coordinates of the point of interest on the left and right images. Once the coordinates of the point of interest on the left and right images are obtained, the three-dimensional coordinates of the point of interest are derived by the principle of triangulation. The above is the principle of deriving the three-dimensional coordinate information of the point of interest from the line-of-sight information.

(Point-of-interest time tracking operation) In the above three-dimensional coordinate derivation method, as a stereo image display method, a time-series stereo image is displayed in a time-series order, and the stereo image is displayed as a moving image. At this time, the observer who observes the stereo image can consciously track the extraction point of the moving object. As described above, a moving image captured in stereo is displayed on a stereoscopic image display device, and a point of interest is tracked while the object is viewed stereoscopically. Is obtained.

(Details of the stereoscopic image display device and the visual line detection device) Next, the stereoscopic image display device 3013 and the visual line detection device 30
14 will be described in detail.

FIG. 6 shows a configuration of a stereoscopic image display device 3013 and a visual line detection device 3014 used in the embodiment of the present invention.

Reference numeral 205 denotes a stereoscopic image display device with a visual axis detection device including the stereoscopic image display device 3013 and the visual axis detection device 3014. The display device is incorporated in a housing mounted on the head. Normally, Head Mount
It is called ed Display (HMD).

First, the display system will be described.

The video input circuit 6003 is connected to the computer 2
The video signal from the terminal 04 is input. LCD display circuit 600
2 displays a stereo image on two left and right LCDs (liquid crystal display elements) 501 and 502, respectively. As described above with reference to FIG. 5, the images on the LCDs 501 and 502 are guided by the prism type eyepieces 503 and 504 as appropriate positions and virtual images on the entire observer.

Next, the gaze detection system will be described.

Reference numeral 6010 denotes an infrared light emitting diode (IRED) for illuminating an eyeball; 6007 and 6008, an optical system for imaging an eyeball;

A line-of-sight detection device 3014 employed in this embodiment.
Has already been implemented and the principles are well known,
Here, the principle of gaze detection will be briefly described with reference to FIGS. The observer's eyeball is illuminated by two infrared light emitting diodes (IRED6010 in FIG. 6) arranged symmetrically with respect to each eyeball, and photographed by the image sensor 6006 via the optical systems 6007 and 6008. FIGS. 7B and 7D are schematic diagrams of an eyeball photographed image when the observer is gazing at the front and gazing at the left. In FIGS. 7B and 7D, black areas are pupil images, and two bright points in the pupil are virtual images of two IREDs reflected by the cornea.

As shown in FIGS. 7A and 7C, the inclination angle a of the optical axis of the eyeball with respect to the horizontal direction is the difference D between the pupil center position C and the corneal curvature center position O in the vertical direction, and the pupil center. It can be determined by the distance CO between the corneal curvature centers.

The position C of the pupil center can be determined by obtaining the center point of the pupil image area by analyzing the image of the eyeball. Further, the position O of the center of curvature of the cornea is represented by IRE
D can be determined from the position of the virtual image due to the corneal reflection. Therefore, the difference D between the center position of the pupil and the center position of the corneal curvature is obtained from the analysis of the captured eyeball image.

On the other hand, the distance C between the pupil center and the corneal curvature center
O is an individual dependent factor. In addition, the optical axis of the eyeball at the time of center gaze usually does not coincide with the axis connecting the center of the eyeball and the front center point in front (direct viewing axis), and has a certain angle with respect to the direct viewing axis. This angle is also a factor that depends on the individual. These individual factors are treated as two correction factors in calculating the line-of-sight angle from the eyeball photographed image. After all,
The horizontal line-of-sight angle a of one eye can be expressed as a function of the horizontal distance dx between the center of the pupil and the center of the corneal curvature, and the correction terms C1 and C2 as shown in Expression (1).

A = arcsin (dx × C1) −C2 (1) Similarly, the vertical line-of-sight angle b of one eye is the vertical distance dy between the pupil center and the corneal curvature center, and the correction terms C3 and C
Equation (2) can be expressed as a function of Eq.

B = arcsin (dy × C3) −C4 (2) The correction terms of C1, C2, C3, and C4 are determined by a correction process performed once for each individual gaze detection action. The correction process is performed by displaying a specific correction point on a screen, causing the observer to gaze at the point, and analyzing the captured eyeball image at that time. Once the correction term is determined, the line-of-sight angle can be calculated at any time from the information of the eyeball photographed image. By performing the above processing on the left and right eyeballs, the line of sight of the left and right eyeballs can be obtained.

In this embodiment, the line-of-sight detection circuit 6005
Only x and dy are calculated. dx and dy are sent to the computer 204 by the line-of-sight data transmission circuit 6004,
Equations (1) and (2) are calculated by a computer program. However, without calculating the formulas (1) and (2) on the computer side, the formula (1),
It goes without saying that the calculation up to the expression (2) may be performed.

In the present embodiment, the data output from the visual line detection device 3014 is hereinafter referred to as visual line data.
That is, for the left eyeball, the horizontal distance between the pupil center and the corneal curvature center and the vertical distance (dx_l, dy_
l), and the horizontal distance and the vertical distance (dx_r, dy) between the pupil center and the corneal curvature center for the right eyeball.
_R) will be referred to as line-of-sight data.

In the present embodiment, the three-dimensional image display device 30
A head-mounted stereoscopic image display (HMD) 205 in which the camera 13 and the eye-gaze detecting device 3014 are integrated is used. In this case, the relative position between the stereoscopic image display and the observer's head position is fixed, and attention is paid with high accuracy. This is because point coordinates can be detected.
However, the present invention is not limited to this. When the observer's head position does not fluctuate with respect to the stereoscopic image display device, or even if it fluctuates, a mechanism for detecting this change amount and a point of interest based on the change amount If a coordinate correcting means is provided, the stereoscopic image display device and the visual line detection device may be separated. For example, a desktop interlaced display and a liquid crystal shutter glasses commonly used as a stereoscopic image display device may be used, or a lenticular type stereoscopic display without glasses may be used, and the gaze detection device is simply attached to the glasses. However, the present invention can be easily realized.

Next, a procedure for performing a three-dimensional motion input by the apparatus having the above configuration will be described with reference to a flowchart of FIG.

As described in the principle of the line of sight detection, it is necessary to determine an individual dependent correction term for the line of sight detection. Therefore, prior to the three-dimensional motion input operation, first, correction processing relating to gaze detection is performed in step S8002.

The computer displays the marker for correction on the three-dimensional image display device 3013, and makes the observer gaze at the marker. At this time, the line-of-sight data dx and dy transmitted from the line-of-sight detection device 3014 are acquired.

The display position of the sign, that is, the line-of-sight angles a and b when the observer gazes at the sign is accurately known, and the line-of-sight data dx and d corresponding to the line-of-sight angles a and b.
The correction term is determined by substituting y into equations (1) and (2). The correction term can be determined by measuring two points per eye, but it is also possible to determine two or more points and determine by the least square method.

Next, in step S8003, the photographed time-series stereo images are stored as digital image data in the secondary storage device 3015 of the computer 204 so that the computer 204 can process them. Hereinafter, this step will be described.

A time-series stereo image photographed by the stereo image photographing device 3010 is recorded on a recording medium 3011 such as a video tape.

First, a left or right eye camera 101,
A method of capturing a time-series image from a recording medium captured by one of the image capturing apparatuses 102 (stereo image capturing apparatus 3010) will be described. The recording medium 3011 is a recording medium reproducing device 30.
12 reproduces a video signal. When the stereo image capturing device 3010 to be used is an analog video camera, the reproduced video signal is an analog signal, and the recorded video is an analog / video signal of the video signal input device 3012.
The data is converted into digital image data in chronological order by a digital conversion mechanism. Such video signal input devices are now commonly available. Stereo image capturing device 3010 to be used
Is a digital video camera, a device that outputs a dedicated digital signal is often provided as the recording medium reproducing device 3012. In this case, the dedicated digital signal is converted into a digital image data as a video signal input device in chronological order. It is sufficient if there is a mechanism for converting to. Such a recording medium reproducing apparatus is usually incorporated in a digital video camera body. A video signal input device for digital signals can be easily obtained as a peripheral device of a computer.

Next, a method of forming a time-series stereo image will be described. Here, of the recording media 3011, the recording medium of the right-eye camera 202 is the recording medium 1, and the left-eye camera 201 is
Is a recording medium 2. A computer 20 converts an image at a certain time from the recording medium 1 into digital image data.
Take in 4. As described above, since the time code is recorded on the recording medium, the time code is read from the recording medium 1, the image having the time code corresponding to this time code is read from the recording medium 2, and the computer 204 is read as digital image data. Secondary storage device 3015
Take in. To capture an image having a specific time code, it is necessary to control the recording medium reproducing device 3012, which is performed by a reproduction control circuit 3016.

Normally, a video tape reproducing apparatus for a broadcast business accepts an external command by serial data,
It is possible to control image reproduction of a specific time code and the like.
In this case, the reproduction control circuit 3016 corresponds to a serial communication control circuit of a computer. Recording medium reproducing device 30
In the case where there is no external control function in 12, the operator may perform the operation manually. If the reproducing device of the recording medium 1 and the reproducing device of the recording medium 2 are controlled to reproduce the same time code image, then a stereo image can be captured by simple frame-by-frame reproduction.

The two digital image data having the same time code captured in this way are thereafter recognized and managed as a set of stereo images at the same time. This management uses a data management function on a secondary storage device that is generally provided in a basic software of a computer called an operating system. That is, a name is generated by combining a character for distinguishing the time-series data group, an integer number corresponding to the time, and a character for distinguishing between an image for the right eye and an image for the left eye. Save. For example, the image for the right eye at time 1, the image for the left eye at time 1, the image for the right eye at time 2, and the image for the left eye at time 2 collected from the video named “test” are “test- 0001-r "," test-000
1-1 ”,“ test-0002-r ”,“ test-
0002-1 "in a file.
The storage file may be subjected to image data compression such as JPEG.

In the above description, the camera 202 for the right eye
Is the recording medium 1, but the recording medium of the left-eye camera 201 may be the recording medium 1, and the same applies to the recording medium 2.

The above description assumes that the secondary storage device 3015 of the computer has a sufficient capacity and can store all necessary time-series stereo images. But,
Even if the secondary storage device does not have a sufficient capacity, the secondary storage device 301 can read a stereo image at a specific time from the recording medium onto the image memory at a sufficient speed.
It is not necessary to accumulate the time-series stereo images in 5. In this case, the stereo image data may be read from the recording medium only when the stereo image to be processed is needed.

After the reading and accumulation of the time-series stereo image in step S8003, a subroutine for deciding from which portion of the captured video a motion input is to be performed is entered. Hereinafter, this subroutine will be described with reference to FIG.

First, a time number N for managing time is initialized to a value “1” (S9002). This initial value may be any value as long as it is within the time range of the read time-series stereo image.

Next, the stereo image corresponding to the time number N is read out and displayed on the stereoscopic image display device (S90).
03). Hereinafter, this operation will be described.

The file of the right-eye image and the left-eye image of the time number N is read from the secondary storage device 3015 of the computer 204. As described in “Reading a time-series stereo image” in step S8003, a file corresponding to the right-eye image and the left-eye image of the time number N can be specified from the saved file name. The data of the specified file is read from the secondary storage device 3015 of the computer 204. The read two images are stored in the image memory 3005. The left and right image data placed on the image memory 3005 are converted into video signals by the video signal output circuit 3006 and output to the stereoscopic image display device 3013. This video signal is obtained by interlacing the left and right images, NTS
The signal is output to the HMD 205 as a C interlace signal.
The HMD 205 receives this interlaced video signal, separates the left and right images, and outputs a left-eye LCD 501 and a right-eye L
A right-eye image and a left-eye image are displayed on the CD 502, respectively. Instead of using the NTSC interlace signal, a video signal dedicated to the present apparatus may be used, and each image may be displayed using independent signal lines for the left and right images.

As described above, the stereoscopic image is displayed on the stereoscopic image display device 3013, and the observer watching the display device is in a state where the above-described fusion operation is possible, that is, a state where the stereoscopic view is possible.

Next, the flow advances to step S9004 to check an input from the operator. If it is determined in step S9005 that the input is a "start request for three-dimensional motion input", the start point determination subroutine is terminated, and the process returns to step S8005 of the main flow in FIG. move on.

In step S9007, the input from the operator is further examined. If the input is "a request to increase the time number by 1", the time number N is increased by "1" in step S9008, and the stereo image of the next time is increased. To return to step S9003. If the input is not “a request to increase the time 1 by a step” (N in step S9007)
o) In step S9009, the input is further examined. If the input is "a request to change the time number to the Mth", M is substituted for the time number N in step S9010, and a stereo image of the time number M is displayed. The process returns to step S9003. Here, M is a numerical value input by the operator. If the input is not the “request to change the time number to the M-th” (No in step S9009), the process returns to step S9004 to wait for an input by the operator.

By the above-described repetitive processing, the operator can select a desired exercise input start point. You will also be given the opportunity to choose a motion extraction point here. That is, the operator can observe the display object until the “request for starting three-dimensional motion input” is given to the computer. Therefore, during this time, the appearance of the object can be observed and a place suitable for motion extraction can be searched. When the motion extraction point is determined, the operator gives an input of “request for starting three-dimensional motion input” to the computer 204 while gazing at the motion extraction point, and enters a three-dimensional motion input process.

When the start point is determined in step S8004, the flow advances to step S8005. In step S8005, a variable T for storing the time at which the stereo image was updated is stored.
The value of the current time is substituted for _last. Thereafter, the control program of the computer repeats the following processing routine until the termination condition is satisfied.

First, an instruction from the operator through the input device of the computer 204 is checked. The input from the input device is checked (S8006). If there is an input, the input data is extracted (S8007), and the process proceeds to step S8008.

In step S8008, if the input is "exercise input termination request", the program is terminated. At this time, three-dimensional motion data described below is
04 in the secondary storage device 3015, and the processing ends. If the input is not “exercise input termination request”, the process proceeds to step S8009.

In step S8009, the current time and "T
_Last + dT ”. Here, dT is a reproduction speed adjustment parameter for reproducing a time-series stereo image as a moving image. Assuming that the desired reproduction speed, that is, the frame rate is N frames / second, dT = 1/1 /
Determined as N. If the current time is greater than “T_last + dT” in the comparison in step S8009,
It is determined that the time to perform the coordinate extraction and the update of the stereo image has arrived, and the process advances to step S8010. Step S800
If the current time is smaller than "T_last + dT" in the comparison of No. 9, it is determined that the standby period is for the reproduction speed adjustment, and the process returns to step S8006 again to check the input from the operator. Here, any other processing may be performed if necessary during the standby period for adjusting the reproduction speed.

In step S8010, line-of-sight data is obtained. The line-of-sight data is periodically transmitted from the line-of-sight detection device 3014 and received by the line-of-sight data input circuit 3007 of the computer 204. Therefore, in order to acquire the line-of-sight data, the latest line-of-sight data received by the line-of-sight data input circuit 3007 may be read. When the line-of-sight data is prepared, the line-of-sight angle is calculated for each of the left and right sides according to Equations (1) and (2). Here, the correction terms of Expressions (1) and (2) have been determined in Step S8002. When the left and right viewing line angles corresponding to the coordinate extraction points are prepared, the flow advances to the next step S8011.

In step S8011, a process of calculating three-dimensional coordinates is performed. The left and right gaze angles obtained in step S8010 are respectively converted into two-dimensional coordinates of the extraction point on the left and right images by the following equation (3).

Xl = L × tan (al) Yl = L × tan (bl) Xr = L × tan (ar) Xr = L × tan (br) (3) where, al, bl, ar, br, Xl , Yl, Xr,
The meanings of Yr and L are as follows.

Al: The rotation angle when the Y-axis of the left eyeball optical axis is used as the rotation axis bl: The rotation angle when the X-axis of the left eyeball optical axis is used as the rotation axis ar: The Y-axis of the right eyeball optical axis is rotated Rotation angle when the axis is the axis br: Rotation angle when the X axis of the right eyeball optical axis is the axis of rotation Xl: X coordinate of the extraction point on the image for the left eye Yl: Y of the extraction point on the image for the left eye Coordinates Xr: X coordinate of extraction point on right-eye image Yr: Y coordinate of extraction point on right-eye image L: distance from eyeball to display surface X-axis is as shown in FIG. , The direction parallel to the plane of the paper and perpendicular to the optical axis F (vertical direction on the paper), and the Y axis is a direction orthogonal to the X axis and the optical axis F ( (A direction perpendicular to the paper surface).

From the set of two-dimensional coordinates (Xl, Yl) and (Xr, Yr) on the left and right images obtained based on the above equation (3), the three-dimensional coordinates (Xc , Y
c, Zc) is calculated based on the following equation (4).

Xc = X1 × t Yc = Y1 × t Zc = L × t (4) Here, t is defined as t = D / (X1−Xr), and D represents the distance between the left and right eyes.

When the three-dimensional coordinates of the extraction point are obtained by the equation (4) in step S8011, the flow advances to step S8012.

In step S8012, step S801 is executed.
Three-dimensional coordinates (Xc, Yc, Zc) of the extraction point obtained in 1
Is added to the three-dimensional motion data as three-dimensional coordinates at a specific time of the motion extraction point.

As shown in FIG. 11A, the three-dimensional motion data has an array structure in which three-dimensional coordinates at a certain time are stored as elements, and the elements are prepared for the total number of times. I have. This array structure is configured on the memory 3020 of the computer 24. Hereinafter, this array structure is referred to as motion array data.

When the three-dimensional coordinates at the N-th time are derived, the N-th element of the motion sequence data is filled and held. When extracting motions at a plurality of motion extraction points, one motion array data may be prepared for each motion extraction point, and a plurality of motion array data may be provided. When the motion is extracted at equal time intervals, the time number is uniquely determined from the position of the element in the motion array, so that time information per element is unnecessary. However, assuming motion extraction at uneven time intervals,
Time number information may be added to one element ((b) of FIG. 11).

At the end of the motion input, when the three-dimensional motion data is to be stored, the contents of the motion sequence data configured on the memory 3020 of the computer 24 are stored in a storage device such as a hard disk.

As described above, in step S8012, the extracted three-dimensional coordinates are added to the three-dimensional motion data. Thereafter, in order to process the stereo image at the next time, the time number N is incremented by "1" in step S8013. Next, in step S8014, the stereo image corresponding to the time number N is read and displayed on the stereoscopic image display device. The processing in step S8014 is exactly the same as the processing described in “Reading and Displaying Stereo Image of Time Number N” (Step S9003 in FIG. 9) in the exercise input start point determination subroutine in step S8004. Then, the description is omitted. When the update of the stereo image is completed in step S8014, step S8005 is performed to repeat the exercise input.
Return to

By performing the above-described processing, three-dimensional motion information can be obtained. The obtained three-dimensional motion information may contain various kinds of noise such as a gaze detection error in addition to the actual three-dimensional motion information of the object. This noise can be removed to some extent by the following method.

In some cases, the characteristics of the motion can be predicted from the laws of physics or the like. In this case, the equations of the predicted motion curve have finite parameters, and matching is performed with the three-dimensional motion information data group by the least squares method or the like. I do. The motion curve obtained from the parameters determined in this way may be adopted as the target three-dimensional motion information. Even when the characteristics of the exercise cannot be predicted, an appropriate higher-order curve including many parameters may be used as the exercise prediction curve.

As described above, according to the first embodiment of the present invention, the three-dimensional coordinates of the motion extraction points are automatically extracted in chronological order, enabling efficient three-dimensional motion input.

<Second Embodiment> In the first embodiment, the transition from the processing of the stereo image at a certain time to the processing of the stereo image at the next time is automatically performed by a computer. Although this method is efficient, there is a problem that the operator cannot keep his / her eyes away from the movement extraction point at all times from the start to the end of the movement input. Therefore, in the second embodiment, the time transition is performed by the operation of the operator. Hereinafter, the operation procedure in the second embodiment will be described with reference to the flowchart of FIG.

In the flowchart of FIG. 10, the same processes as those in FIG. 8 are denoted by the same step numbers, and detailed description thereof will be omitted.

Steps S8002 to S800
Preparation for exercise input is performed by the processing up to 4, and then
In steps S10005 and S10006, input from the operator is waited, and if there is an input, the input content is inspected. In the next step S8008, if the input is “exercise input termination request”, the program is terminated. If the input is not “exercise input termination request”, step S1 is executed.
Proceed to 0007. In step S10007, the input is further examined. If the input is "coordinate extraction request", the process proceeds to step S8010 and the subsequent steps to perform coordinate extraction and update the stereo image. If the input is not “coordinate extraction request”, the process returns to step S10005 to monitor the input.

According to the method described above, the three-dimensional coordinates of the motion extraction points are semi-automatically extracted in chronological order based on the movement of the operator's line of sight, and efficient three-dimensional motion input becomes possible.

<Modification 1> In the first or second embodiment, the three-dimensional motion of an object is referred to as a coordinate system based on the position and orientation of a photographing camera, that is, in the CG field, a view space or a camera space. I am looking for something that can be broken. However, in actual photographing, both the photographing target and the camera move, and there is also a request that the movement of the photographing target is desired to be obtained with respect to a stationary coordinate system such as the ground. In such a case, the three-dimensional motion extraction in the first or second embodiment is performed on both the shooting target and the extraction points of the stationary coordinate system, for example, both trees and buildings that are stationary with respect to the ground. Just do it. Then, by subtracting the three-dimensional coordinates of the extraction point in the stationary coordinate system from the three-dimensional coordinates of the imaging target, the three-dimensional motion of the imaging target can be obtained with respect to the stationary coordinate system.

<Modification 2> The modification 1 is based on the idea of removing the influence of camera movement from the movement of the target object. It may be used to improve the accuracy of inputting three-dimensional coordinates of a stationary object, that is, inputting the three-dimensional shape of a stationary object. Hereinafter, this method will be described.

At the time of stereo shooting, a camera is moved according to a known rule to shoot an object in stereo. For example, rotation photography is performed at a known rotation speed around a known point near the target object. As a result, an image of an object rotating at a constant angular velocity can be obtained. Thereafter, the three-dimensional motion of the extraction point is extracted using the same method as in the first or second embodiment. The three-dimensional movement of the extraction point can be accurately represented, ie, predicted, as a function of time, with the extraction point position at one time being the only unknown, and the three-dimensional coordinates as a function of time. Therefore, the least squares method or the like is applied to the three-dimensional motion information data group obtained by the first or second embodiment, and an unknown point position at one time is obtained. As a result, three-dimensional coordinate information obtained at a certain time, that is, with higher accuracy than information obtained from a set of stereo images is obtained. Also, the motion given to the camera need not be known. In that case, an appropriate higher-order curve or the like including many parameters is used as the motion curve of the object, and the least-squares method or the like is applied to the three-dimensional motion information data group to determine the motion curve. Extraction point three-dimensional coordinates at a certain time obtained from the determined motion curve, as in the case of giving a known motion,
It has three-dimensional coordinate information with higher accuracy than information obtained simply from a set of stereo images.

The improvement of the accuracy of the extracted coordinates by a plurality of photographing of the same point can be achieved by using a stereo still image photographed from a plurality of directions in a conventional three-dimensional shape input method using a stereo still image. Are equivalent.

However, the feature of the present embodiment is that information for shape extraction is obtained as a three-dimensional motion, which is a great advantage. That is, by connecting different stereo images with continuous images, continuous tracking based on the line of sight of the extraction points becomes possible, and the correspondence of the extraction points between the different stereo images can be automatically given. That is, when a plurality of discontinuous stereo images are used as still images instead of continuous time-series images as in the related art,
In addition to indicating corresponding points of the left and right images in a set of left and right stereo images, corresponding points must be indicated between different stereo images. On the other hand, according to the present embodiment, the corresponding point indication of the left and right images in a set of left and right stereo images is automated by the fusion operation of the human, and the corresponding point indication between different stereo images is also described above. It is automated by a human target point time tracking operation.

Since the camera and the object are spatially relative to each other, the intentional movement of the camera may be given to the object. Obviously, this has the same effect.

<Third Embodiment> In the present technology, the method for detecting corresponding points according to the present invention has a natural pattern matching based on human image recognition, that is, a fusion operation is superior to a pattern matching based on correlation calculation by a computer. It is based on the point. The comparison criteria are accuracy, processing time,
It is versatile in terms of operability, etc., and will also be affected by future improvements in computer performance. For this reason, there may be situations where the use of correlation calculation by a computer is determined to be superior to the method based on human fusion operation for detecting corresponding points in a stereo image.

However, the present invention is effective even in such a situation, and the present invention can be used together with the correlation calculation of a computer. That is, the present invention is used as pre-processing of correlation calculation by a computer.

Specifically, step S in FIG. 8 or FIG.
The correlation calculation is applied to the processing in 8011. Based on the set of corresponding points obtained in step S8010, the area for performing the correlation calculation is narrowed down, and the computer calculates the correlation for this small area. In this way, the calculation time can be reduced as compared with simply performing the correlation calculation.
Also, when performing a correlation calculation, it is necessary to manually specify an extraction point on one of the left and right images at one time on the time axis. However, since this operation can be specified by eye-gaze input, it is efficient.

[0117]

According to the present invention, when extracting three-dimensional motion information from a real image by a stereo shooting method, the amount of work and the time required for extracting a time series of three-dimensional coordinates can be greatly reduced. Therefore, it is easy to use the stereo imaging method having the merit of eliminating the need for a sensor as the three-dimensional motion input method, and it is possible to broaden the imaging target of the three-dimensional motion input.

Further, by using the present invention for inputting the shape of a stationary object, it is possible to easily improve the accuracy of shape display. Further, by obtaining three-dimensional coordinates with reference to a stationary object in a captured image, the degree of freedom at the time of capturing is improved. Moreover, the burden on the observer can be reduced by the observer freely instructing the timing of the motion input.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating three-dimensional coordinate extraction using stereo image capturing.

FIG. 2 is a conceptual diagram illustrating a device configuration of an exercise input system according to an embodiment of the present invention.

FIG. 3 is a configuration diagram of a computer and peripheral devices used in the embodiment of the present invention.

FIG. 4 is a diagram illustrating the principle of stereoscopic viewing.

FIG. 5 is a diagram for explaining a method of separating and inputting left and right images.

FIG. 6 is a block diagram illustrating the configuration of a head-mounted stereoscopic image display device with a visual axis detection device.

FIG. 7 is a diagram illustrating the principle of gaze detection.

FIG. 8 is a flowchart illustrating a processing procedure of a three-dimensional motion input according to the first embodiment of the present invention.

FIG. 9 is a flowchart illustrating a subroutine for determining an exercise input start point according to the first embodiment.

FIG. 10 is a flowchart illustrating a processing procedure of a three-dimensional motion input according to the second embodiment of the present invention.

FIG. 11 is a diagram illustrating a structure of three-dimensional motion data.

FIG. 12 is a diagram illustrating a spatial-temporal corresponding point designating operation of a stereo imaging method in the related art.

[Description of Signs] 101 Left-eye camera 102 Right-eye camera 103 Left image 104 Right image 105 Representative point in left image 106 Representative point in right image 201 Left-eye camera 202 Right-eye camera 203a Moving object 203b at time 1 Moving object at time 2 204 Computer 205 Stereoscopic image display device with visual line detection device 206a Stereoscopic 3D image of moving object at time 1 206b Stereoscopic 3D image of moving object at time 2 207 3D motion information 3002 Video signal input circuit 3003 Main Storage device 3004 CPU 3005 Image memory 3006 Video signal output circuit 3007 Eye-gaze data input circuit 3008 Mouse 3009 Keyboard 3010 Stereo image photographing device 3011 Recording medium 3012 Recording medium reproducing device 3013 Stereoscopic image display device 3014 Eye-gaze detection device 3015 Secondary storage device 3020 Memory 402 Point of interest 403 Real object 405 Display image for right eye 406 Display image for left eye 407 Point of interest on virtual object 501, 502 LCD 503, 504 Prism eyepiece 505 LCD display surface virtual image position 6002 LCD display circuit 6003 Video input circuit 6004 Eye-gaze data transmission circuit 6005 Eye-gaze detection circuit 6006 Eyeball image sensor 6007 Eyeball lens 6008 Eyeball lens 6010 Infrared light emitting diode IRED

Continued on front page F-term (reference) 2F065 AA04 AA09 AA39 CC16 DD06 FF05 FF41 GG07 GG21 JJ03 JJ05 JJ26 QQ24 QQ41 SS13 5B057 CA08 CA13 CA16 CB08 CB13 CB16 CD14 CH08 DA07 DA20 DB03 DB09 DC34 5B087 BC13 DJ13 BC13

Claims (21)

[Claims] An imaging step of capturing a pair of moving images having parallax; and a pair of the moving images captured in the imaging step.
A display step of performing stereo display so as to be stereoscopically viewed, a gaze detection step of detecting a gaze direction of an observer observing an image stereo-displayed in the display step, and a gaze direction detected in the gaze detection step. A corresponding point detecting step of detecting a corresponding point of the pair of moving images; a calculating step of calculating three-dimensional coordinates of a point on the moving image corresponding to the corresponding point detected by the corresponding point detecting step; Storing the three-dimensional coordinates obtained in the calculation step in association with time. 2. A reading step of reading a pair of moving images having parallax from a recording medium, and a display for stereoscopically displaying the pair of moving images read from the recording medium in the reading step so as to be stereoscopically viewed. A step, a line-of-sight detection step of detecting a line-of-sight direction of an observer who observes the image stereo-displayed in the display step, and a corresponding point of the pair of moving images based on the line-of-sight direction detected in the line-of-sight detection step. A corresponding point detecting step of detecting, a calculating step of calculating three-dimensional coordinates of a point on a moving image corresponding to the corresponding point detected in the corresponding point detecting step, and a three-dimensional coordinate obtained by the calculating step. A storage step of storing in association with time. 3. The three-dimensional motion input method according to claim 1, wherein in the photographing step, a moving image is photographed with the photographing position and direction fixed. 4. The three-dimensional motion input method according to claim 1, wherein in the photographing step, a moving image is photographed while moving a photographing position on a predetermined trajectory. 5. In the corresponding point detecting step, respective corresponding points of a point on the stationary object in the moving image and a point of the moving object in the moving image are detected. 3. The three-dimensional coordinates of a point on the moving object based on the point (3) are calculated.
3. The three-dimensional motion input method according to 1. 6. The display step, wherein at each predetermined time, a pair of images at one time of the pair of moving images is updated to an image at the next time and displayed, and in the calculating step,
The three-dimensional motion input method according to claim 1, wherein three-dimensional coordinates of a point on a moving image are calculated at the predetermined time interval. 7. The display step, wherein a pair of images at one time of the pair of moving images is updated to an image at the next time based on an external instruction, and the calculation step includes: The three-dimensional motion input method according to claim 1, wherein three-dimensional coordinates of a point on the pair of images updated in the display step are calculated. 8. In the calculating step, a two-dimensional correlation operation is performed on a two-dimensional area near the corresponding point obtained in the corresponding point detecting step in the pair of moving images, and based on the obtained correlation value, 8. The method according to claim 1, wherein a new corresponding point on the pair of moving images is obtained, and three-dimensional coordinates of the point on the moving image are calculated based on the newly obtained corresponding points. 3D motion input method. 9. The eye-gaze detecting step includes irradiating the observer's eyeball with invisible light, and determining the observer's eyeball from the relative relationship between the observer's eyeball pupil center position and the virtual image position of the invisible light from the corneal reflecting surface. The three-dimensional motion input method according to any one of claims 1 to 8, wherein a rotation angle of the eyeball is obtained, and a gaze direction is detected. 10. A photographing means for photographing a pair of moving images having parallax, a display means for stereoscopically displaying the pair of moving images photographed by the photographing means so as to be viewed stereoscopically, and the display means. Gaze detection means for detecting a gaze direction of an observer observing an image displayed in stereo; and corresponding point detection means for detecting a corresponding point of the pair of moving images based on the gaze direction detected by the gaze detection means Calculating means for calculating three-dimensional coordinates of a point on a moving image corresponding to the corresponding point detected by the corresponding point detecting means; and storing the three-dimensional coordinates obtained by the calculating means in association with time. A three-dimensional motion input system, comprising: a storage unit. 11. A reading means for reading a pair of moving images having parallax from a recording medium, and a display for stereoscopically displaying the pair of moving images read from the recording medium by the reading means so as to be stereoscopically viewed. Means, a gaze detecting means for detecting a gaze direction of an observer observing an image stereo-displayed on the display means, and a corresponding point of the pair of moving images based on the gaze direction detected by the gaze detection means. Corresponding point detecting means for detecting the three-dimensional coordinates of a point on a moving image corresponding to the corresponding point detected by the corresponding point detecting means, and the three-dimensional coordinates obtained by the calculating means Storage means for storing in association with time. 12. The three-dimensional motion input system according to claim 10, wherein said photographing means photographs a moving image with the photographing position and direction fixed. 13. The three-dimensional motion input system according to claim 10, wherein said photographing means photographs a moving image while moving on a predetermined trajectory. 14. The corresponding point detecting means detects respective corresponding points of a point on a still object in the moving image and a point of a moving object in the moving image, and the calculating means
The three-dimensional coordinates of a point on a moving object with reference to the point on the stationary object are calculated.
3. The three-dimensional motion input system according to item 1. 15. The display means updates a pair of images at one time of the pair of moving images to an image at the next time at predetermined time intervals, and displays the pair of images at a predetermined time. The three-dimensional motion input system according to any one of claims 10 to 14, wherein three-dimensional coordinates of points on the moving image are calculated at intervals. 16. The display means updates a pair of images at one time of the pair of moving images to an image at the next time based on an external instruction and displays the pair of images, respectively, 15. The three-dimensional motion input system according to claim 10, wherein three-dimensional coordinates of a point on the pair of images updated by the display unit are calculated. 17. The calculation unit performs a two-dimensional correlation operation on a two-dimensional area near a corresponding point obtained by the corresponding point detection unit in the pair of moving images, and performs the two-dimensional correlation operation based on the obtained correlation value. 17. The method according to claim 10, wherein a new corresponding point on the pair of moving images is obtained, and three-dimensional coordinates of the point on the moving image are calculated based on the newly obtained corresponding points. 3D motion input system. 18. The eye-gaze detecting means has a light source for irradiating an invisible light to an observer's eyeball. The three-dimensional motion input system according to any one of claims 10 to 17, wherein a rotation angle of an observer's eyeball is obtained, and a gaze direction is detected. 19. The stereoscopic image display device according to claim 10, wherein said display means is a head-mounted stereoscopic image display device capable of fixing a relative positional relationship between a viewer's head and a display surface.
19. The three-dimensional motion input system according to any one of claims 18 to 18. 20. The three-dimensional motion input system according to claim 19, wherein said head-mounted image display device has said line-of-sight detecting means. 21. A stereoscopic image display device having a mechanism for detecting a relative position of an observer's head with respect to the display device, wherein the relative position of the observer's head with respect to the stereoscopic image display device is provided. 21. The three-dimensional motion input system according to claim 10, wherein a change is detected, and a line-of-sight direction in said line-of-sight detecting means is corrected based on the detected relative position change.