JP4581204B2 - POSITION ANALYSIS DEVICE, POSITION ANALYSIS METHOD, AND ENTERTAINMENT DEVICE - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention detects the relative position of the other device with respect to one device. for position analysis Equipment and position analysis The present invention relates to a method and an entertainment apparatus, and particularly detects a relative position by detecting and analyzing an identifier. for position analysis Equipment and position analysis The present invention relates to a method and an entertainment apparatus to which these are applied.
[0002]
[Prior art]
In recent years, with the development of electronic game technology, games that only require operation input by a combination of several input buttons, or operation input using a so-called joystick or the like, satisfy the user's requirements or increase the user's interest. It has become difficult to acquire.
[0003]
Therefore, for example, an input device imitating a fishing tool is used for a fishing game, an input device imitating a musical instrument is used for playing a musical instrument, and a real gun is imitated for a shooting game. The game device is devised so that the game can be performed by a more realistic operation by using the input device. In addition, there is a game device in which the user moves the body while watching the screen.
[0004]
In this case, instead of detecting the movement of the whole body, the movement of the user's body is input as a partial movement such as a step by using an input device that detects when a predetermined position is touched. ing.
[0005]
In order for a user to input a game by performing an action closer to a realistic action, it is considered that a user's movement is detected and used as an input signal. For example, a game player (user) is imaged by an imaging device, the movement of the game player (user) is analyzed based on the captured image, and the analysis result is reflected in the input of the game.
[0006]
A system for detecting the movement of a user or the position of a predetermined part in a non-contact manner has already been realized. As such a position detection system, for example, a scanning line of a cathode ray tube is exposed with a beam and detected by a light emitting element. In some cases, the position is detected using infrared rays or ultrasonic waves.
[0007]
[Problems to be solved by the invention]
However, when detecting the position of the target part in a non-contact manner, in particular, when using infrared rays, in order to identify the position by detecting the interference fringes by receiving infrared rays from a plurality of infrared ray generators installed in advance. If it is not used at a close distance, the position detection accuracy is lowered. Moreover, since it is easy to receive interference from other devices using infrared rays, there is a problem that the position detection accuracy is lowered particularly when the same devices are used side by side. In addition, when ultrasonic waves are used, the reflected waves interfere with each other in the same manner as when infrared rays are used to determine the position and movement by measuring the reflected waves received by a plurality of ultrasonic sensors installed in advance. Under such conditions, the position detection accuracy is lowered.
[0008]
In addition, a game device in which a user shoots at a target displayed on the screen employs a method of detecting the position on the screen by exposing the scanning line of the cathode ray tube with a beam and detecting it with a light receiving element. However, it is necessary to flash a part or the entire area of the screen every detection operation. For this reason, it can only be applied to an application that does not feel uncomfortable even if the screen is flashed for each position detection operation. In addition, position detection using this method is effective only for a display device having scanning lines, and cannot be applied to an image projected by a projector or an image displayed on a liquid crystal monitor. there were.
[0009]
Therefore, a position detecting device for solving such inconvenience is disclosed in Japanese Patent Laid-Open No. 6-301475. However, when the position detection device disclosed here is applied to a shooting game, the inclination of the imaging device as a gun-type controller is not detected, so unless the user keeps the imaging device as a gun-type controller accurately level. An error occurs between the aim aimed by the user and the detected position.
[0010]
Japanese Patent Application Laid-Open No. 11-305935 discloses a position detection device using an imaging device. However, since an infrared light emitting element is used as an identifier, for example, the position detection device is used as a home display. It is difficult to apply it, and it is limited to use as a game machine with a large-sized case for business use. Further, since this position detection device does not include an infrared cut filter, it cannot be used as a normal camera even if the image pickup device is detached, which is inconvenient.
[0011]
As described above, as a conventional user interface, there are concepts such as a GUI (Graphical User Interface) using a contact type interface and a desktop metaphor, but there are various problems as described above.
[0012]
Therefore, the present invention has been proposed in view of such a conventional situation, and is a position as a new user interface by non-contact image input capable of high-precision motion detection corresponding to all kinds of display devices. analysis Equipment and position analysis Method and such position analysis An object is to provide an entertainment apparatus to which the method is applied.
[0013]
[Means for Solving the Problems]
The position analysis apparatus according to the present invention is an analysis means for analyzing a position of the image pickup means relative to the display means using an identifier picked up by an image pickup means for picking up an identifier displayed on the display means. Storage means for storing the position of a predetermined point within the imaging range imaged in the past; With , The analysis means is , Using images taken by multiple exposure by the imaging means By analogizing the temporal change between the plurality of identifiers displayed in one image as a clue that the position change of the predetermined point in the past stored in the storage means is continuous, By analyzing the position, the above object is achieved.
[0014]
In the position analysis method according to the present invention, the analysis step of analyzing the position of the imaging unit with respect to the display unit using the identifier imaged by the imaging unit that images the identifier displayed on the display unit. And a storing step for storing the position of a predetermined point in the imaging range captured in the past in the storage means; With , In the above analysis process , Using images taken by multiple exposure by the imaging means By analogizing the temporal change between the plurality of identifiers displayed in one image as a clue that the position change of the predetermined point in the past stored in the storage means is continuous, By analyzing the position, the above object is achieved.
[0015]
In addition, the entertainment device according to the present invention includes an analysis unit that analyzes a position of the imaging unit with respect to the display unit using an identifier captured by an imaging unit that captures an identifier displayed on the display unit; Storage means for storing the position of a predetermined point in the imaging range captured in the past; Control means for controlling the program by converting the position information analyzed by the analysis means into control information, and the analysis means uses an image taken by the exposure means by the imaging means. By analogizing the temporal change between the plurality of identifiers displayed in one image as a clue that the position change of the predetermined point in the past stored in the storage means is continuous, The above object is achieved by analyzing the position.
[0016]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
[0017]
The position detection system 1 shown as the first embodiment of the present invention displays an identifier on a display device, captures a display image including the identifier with an imaging device as an operation input device, and analyzes the captured image in an analysis unit. A position detection system that detects a positional relationship between a display device and an imaging device and an orientation of the imaging device with respect to the display device by non-contact by performing analysis, for example, a predetermined position or a predetermined position of an image displayed on the display device It is a user interface which can perform operation input by performing selection operation | movement with the imaging device as an operation input device facing with respect to this image.
[0018]
A position detection system 1 is shown in FIG. The position detection system 1 includes a game housing 10 having a display unit 12 and a speaker 13 which will be described later, and a controller type imaging device 11 imitating a gun.
[0019]
An animation image or the like that is the content of the game is output to the display unit 12, and sound effects, music, and the like are reproduced from the speaker 13 in accordance with the image displayed on the display unit 12. In the position detection system 1 shown in FIG. 1, a background image, an identifier 14, a target 15, and an aiming point 16 are displayed on the display unit 12 although details are not shown.
[0020]
As described above, the position detection system 1 uses the controller-type imaging device 11 as an operation input device imitating the gun of the user 1000, for example, and moves the aiming point 16 toward the target 15 displayed on the display unit 12. It constitutes a game device that shoots in a fixed manner.
[0021]
The controller type image pickup device 11 is an image pickup camera including a CCD (Charge Coupled Device) as an image pickup element, and is displayed on the display unit 12 when the user 1000 faces the display unit 12 of the game housing 10. The surrounding environment including the image is captured, and the image in the imaging range 17 is transmitted to the game housing 10. At this time, the center of the imaging range 17 captured by the controller type imaging device 11 is displayed as a sighting point 16 at a corresponding location on the display unit 12.
[0022]
Next, a specific circuit configuration of the position detection system 1 as the game apparatus is shown in FIG.
[0023]
As shown in FIG. 2, the game case 10 in the position detection system 1 includes a lens unit 20, a CCD (Charge Coupled Device) unit or a CCD imager 21 (Imager) 21, S / H (Sampling / Hold unit 22, AGC (Auto Gain Controller) unit 23, A / D (Analog / Digital) unit 24, signal processing unit 25, TG (Timing Generator) unit 26, imaging drive unit 27, imaging control unit 28, SDRAM ( Synchronous Dynamic Random Access Memory) 29, main memory 30, mechanical I / F (Interface) 31, game control unit 32, display control unit 33, ROM (Read Only Memory) 34, image processing unit 35, and frame memory 36 ing.
[0024]
In the position detection system 1 having such a configuration, a CCD (Charge Coupled Device) unit or a CCD imager 21, an S / H unit 22, an AGC unit 23, an A / D unit 25, a signal processing unit 25, and a TG unit. 26, the imaging drive unit 27, the imaging control unit 28, and the SDRAM 29 constitute the controller type imaging device 11, and the game control unit 32 compares the captured images output from the controller type imaging device 11 to detect motion. It functions as a motion detection means.
[0025]
Specifically, the controller type imaging device 11 has a function of performing imaging at an arbitrary timing by an asynchronous shutter, and a function of performing imaging by multiple exposure shooting that performs a plurality of imaging within one cycle of a synchronization signal for imaging. And have. Details of these functions will be described later.
[0026]
Further, the game control unit 32 detects a motion by comparing an image obtained by photographing the display unit 12 at a certain time with a photographed image after the operation of the game player (user) after a predetermined time has elapsed. The game control unit 32 also has a function as a control unit that executes a program based on the detected motion detection result.
[0027]
In such a position detection system 1, a captured image input through the lens unit 10 is photoelectrically converted by the CCD unit 11 that is a solid-state imaging device. Here, the image pickup device is not limited to the CCD, and other image pickup devices such as a CMOS sensor may be used. The image signal photoelectrically converted by the CCD unit 11 is sampled and held by the S / H unit 22, and then the gain is adjusted by the AGC unit 23 and input to the A / D unit 25. At this time, the readout timing of the image signal from the CCD unit 11 and the processing timing in the S / H unit 22 are controlled by the readout timing signal from the TG unit 26.
[0028]
In the A / D unit 25, the input imaging signal is quantized. The image information quantized by the A / D unit 25 is subjected to general camera signal processing by the signal processing unit 25 and stored in the SDRAM 29 which is an information storage unit. Here, the information storage unit in which the image information is stored is not limited to the SDRAM 29, and may be another volatile storage unit. The image stored in the SDRAM 29 is streamed by the imaging control unit 28 in response to a request from the game control unit 32 and transferred to the main memory 30 via the game control unit 32.
[0029]
The imaging control unit 28 controls each part of the position detection system 1 in an integrated manner. The imaging control unit 28 controls the imaging driving unit 17 to drive the lens unit 10 and controls the aperture, zoom, focus, and the like.
[0030]
The game control unit 32 performs motion detection by performing image processing on the image transferred from the SDRAM 29 as a stream. Here, the game control unit 32 and the imaging control unit 28 may be configured as the same processor.
[0031]
The motion detection is performed by, for example, a multiple exposure photographing technique disclosed in JP-A-9-247556 described later. Details of motion detection using the multiple exposure imaging technique will be described later.
[0032]
The game control unit 32 performs image processing on the transferred image and detects a predetermined feature point. That is, the identifier 14 displayed at a predetermined position on the display unit is extracted, and the center of the imaging range 17 captured by the controller type imaging device 11 is displayed as a sighting point 16 at a corresponding location on the display unit 12.
[0033]
Next, the game control unit 32 stores the coordinates of the identifier 14 extracted as described above. And the game control part 32 repeats the process mentioned above about the image transferred one after another, ie, the image imaged by the controller type imaging device 11, and the image transferred sequentially.
[0034]
The game control unit 32 performs motion detection of the identifier 14 by statistically analyzing the coordinates of the identifier 14 in time series. Further, the motion of the identifier 14 is corrected by moving object prediction.
[0035]
The detected movement of the identifier 14 and the position of the aiming point 16 are treated as a control signal together with a control signal from the mechanical I / F unit 31 and a trigger signal at the time of shooting, and used as a user input signal to the game. The game control unit 32 executes the game based on the control signal as the input signal, and the target, the aim, etc. depending on the aim position and the trigger are displayed on the display unit 12.
[0036]
That is, the game control unit 32 detects the movement of the identifier 14 and includes the movement detection result in the control signal, whereby the movement of the controller type imaging device 11 with respect to the game housing 10 is displayed on the display unit 12. In particular, in the embodiment according to the present invention, it is reflected in the movement of the aiming point 16.
[0037]
When detecting the movement of the aiming point 16, as shown in FIG. 3, a signboard or a voice guide for prompting the user to perform a prescribed operation is provided, or as shown in FIG. By performing a display for prompting a predetermined operation on the display unit 12 synchronously, and performing a calibration operation for detecting the reference relative position of the identifier 14 in advance, the position can be detected more easily.
[0038]
The ROM 34 stores a program for executing the game. The image processing unit 35 performs image processing, and performs, for example, chroma key processing.
[0039]
Subsequently, the controller type imaging device 11 in the position detection system 1 images the environment around the game housing 10 including the display unit 12, and the display unit is displayed according to the orientation of the controller type imaging device 11 with respect to the display unit 12. The process of displaying the aiming point 16 on the screen 12 will be described with reference to FIG.
[0040]
The imaging control unit 28 constantly monitors and detects imaging conditions. In step S1, the imaging control unit 28 determines whether the imaging conditions are appropriate.
[0041]
If the imaging control unit 28 determines that the imaging conditions are not appropriate in step S1, the imaging control unit 28 proceeds to step S2, and the imaging driving unit 27 that controls the iris (Iris), zoom (Zoom), and focus (Focus), and the TG unit 26. Then, the AGC unit 23 and the signal processing unit 25 are immediately controlled to adjust the imaging condition.
[0042]
If it is determined in step S1 that the imaging conditions are appropriate, the process proceeds to step S3 to determine whether or not a motion detection request has been issued.
[0043]
If it is not required to perform motion detection in step S3, the process of step S1 is repeated. On the other hand, when it is requested to perform motion detection, the process proceeds to step S4, where the controller type imaging device 11 acquires an image of the imaging range 17 and outputs it to the game housing 10, and the process of step S1 is performed. repeat.
[0044]
At this time, a process in which the game control unit 32 in the game housing 10 performs motion detection based on an image from the controller type imaging device 11 is shown in FIG.
[0045]
In step S10, the game control unit 32 determines whether an image has been input. If an image is input, the identifier 14 is extracted from the image information in step S11.
[0046]
Subsequently, in step S12, the game control unit 32 stores the coordinates of the extracted identifier with respect to the angle of view, and corrects the direction / angle and the aiming point 16 with respect to the display unit 12 of the controller type imaging device 11 to be described later. In this case, the position is predicted from the reference relative position of the identifier, and the position of the invisible identifier is predicted.
[0047]
Here, the concept for predicting the position of the identifier 14 will be described with reference to FIG. Assuming that the reference relative position as a reference shown in FIG. 7A is always similar to the imaged identifier 14 shown in FIG. 7B, the imaged identifier is as shown in FIG. 7C. , It is predicted that the identifier C exists at the position shown in FIG.
[0048]
In step S <b> 13, the game control unit 32 detects the position of the aiming point 16 from the relative positions of a plurality of identifiers including the identifier supplemented by prediction.
[0049]
In step S14, the detected position of the aiming point 16 is further corrected by moving object prediction, noise is removed, and it is reflected in the input of the game as a control signal.
[0050]
As described above, the position detection system 1 illustrated in FIG. 1 indicates that when the user pulls the trigger of the controller type imaging device 11 as an input device, the image information and the trigger are pulled from the controller type imaging device 11. Trigger information is transmitted to the game housing 10. At this time, the game control unit 32 recognizes the identifier 14 from the image information, and analyzes to which position of the display unit 12 the controller type imaging device 11 is directed from the relative positions of the plurality of identifiers. The analyzed result is sent to the game control unit 32 together with the trigger information, converted into a control signal and displayed as the aiming point 16, and used as an input for the game.
[0051]
Therefore, the position detection system 1 shown as the first embodiment detects the identifier 14 displayed together with the background image on the display unit 12 from the image of the shooting range 17 shot by the controller type imaging device 11. The aiming point 16 can be displayed according to the coordinates of the identifier 14.
[0052]
As a result, the position detection system 1 uses the movement of the game player (user) and the actual shooting action as input information, and can reflect the action of the real world in a virtual world such as a game.
[0053]
Next, a second embodiment of the present invention will be described. A position detection system 2 shown as a second embodiment is an application of the position detection system according to the present invention to a home game device. As shown in FIG. 8, the position detection system 2 includes a monitor 40, a game device 50, and a controller type imaging device 60.
[0054]
Here, the monitor 40 is an image display means, and is usually a television receiver (hereinafter referred to as a television) capable of inputting video and audio from the outside. The game device 50 is a home game device in which the business game housing 10 as shown in the first embodiment is miniaturized, and includes processing means for executing a game according to a predetermined program. The controller-type imaging device 60 is an input device and is provided with other input means such as operation buttons necessary for game input, but is basically an imaging device and is used as a game input device. It can be used as a single imaging device like a normal CCD camera device without stopping to use.
[0055]
For example, the game device 50 and the controller-type imaging device 60 can transmit and receive data by wireless communication or wired communication. The monitor 40, the game device 50, and the controller type imaging device 60 are connected to each other as schematically shown in FIG.
[0056]
In the position detection system according to the second embodiment, the monitor 40, the game device 50, and the controller type imaging device 60 perform the same processing as the processing of the position detection system 1 described in the first embodiment. be able to. That is, in the second embodiment, the monitor 40 corresponds to the display unit 12 of the position detection system 1, and the game apparatus 50 is in the game housing 10 that performs processing such as motion detection in the position detection system 1 described above. It corresponds to the component of.
[0057]
A circuit configuration of the position detection system 2 is shown in FIG. As described above, since the position detection system 2 shown as the second embodiment performs the same processing as the position detection system 1 of the first embodiment, basically, the position detection system 2 according to the first embodiment. The detection system 1 can be a common component. Therefore, the same components as those in the position detection system 1 are denoted by the same reference numerals and detailed description thereof is omitted.
[0058]
As a matter of course, even components that are common to the position detection system 1 of the first embodiment can be appropriately changed so as to be adapted to the position detection system 2 of the second embodiment. Needless to say, it can be done.
[0059]
The position detection system 2 basically performs the same processing as the position detection system 1 shown as the first embodiment. In the controller type imaging device 60, a captured image input via the lens unit 10 is photoelectrically converted by the CCD unit 11.
The image signal photoelectrically converted by the CCD unit 11 is sampled by the sample / hold in the S / H unit 22, subsequently gain-adjusted by the AGC unit 23, and input to the A / D unit 25. Here, the readout timing of the image signal from the CCD unit 11 and the processing timing in the S / H unit 22 are controlled by the readout timing signal from the TG unit 26.
[0060]
In the A / D unit 25, the input imaging signal is quantized. The image information quantized by the A / D unit 25 is subjected to general camera signal processing by the signal processing unit 25 and stored in the SDRAM 29 serving as an information storage unit. The images stored in the SDRAM 29 are streamed by the imaging control unit 28 in response to a request from the game control unit 32 and transferred to the main memory 30 via the game control unit 32.
[0061]
The controller-type imaging device 60 includes an I / F (interface) unit 61, and performs data communication with the game device 50 via the I / F unit 61. The corresponding game device 50 includes an I / F unit 51, and performs data communication with the I / F unit 61 of the controller type imaging device 60 via the I / F unit 51. The I / F 61 and the I / F unit 51 are configured by communication means capable of transmitting and receiving data by, for example, wireless communication or wired communication.
[0062]
In the game apparatus 50, the game control unit 32 performs image processing on the image transferred as a stream from the SDRAM 29, and performs motion detection. About the motion detection by the game control part 32, it is performed by the process similar to the position detection system 1 of 1st Embodiment. The detected result is treated as a control signal together with a control signal from the mechanical I / F unit 31, and is reflected in input information for the game.
[0063]
The game control unit 32 executes a game based on a control signal including input information, and aims through the monitor I / F unit 52 depending on the movement of the controller-type imaging device 60 as an input device moved by the user. Point 16 is displayed on the monitor 40.
[0064]
At this time, the controller type imaging device 60 in the position detection system 2 images the surrounding environment of the game housing 40 including the display unit 12, and the display unit 12 according to the orientation of the controller type imaging device 60 with respect to the display unit 12. The process of displaying the aiming point 16 on the screen will be described with reference to FIG.
[0065]
The imaging control unit 28 constantly monitors and detects imaging conditions. In step S21, the imaging control unit 28 determines whether the imaging conditions are appropriate.
[0066]
If the imaging control unit 28 determines that the imaging conditions are not appropriate in step S21, the imaging control unit 28 proceeds to step S22, and the imaging drive unit 27 that controls the aperture (Iris), zoom (Zoom), and focus (Focus), and the TG unit 26. Then, the AGC unit 23 and the signal processing unit 25 are immediately controlled to adjust the imaging condition.
[0067]
If it is determined in step S21 that the imaging conditions are appropriate, the process proceeds to step S23, where it is determined whether a motion detection request has been issued.
[0068]
In step S23, when there is no motion detection request, the process returns to step S21. When it is requested to perform motion detection, the process proceeds to step S24, and the imaging control unit 28 in the controller type imaging device 60 acquires the image of the imaging range 17, and compresses it in a predetermined format.
[0069]
In step S <b> 25, the imaging control unit 28 outputs an image compressed in a predetermined format to the game device 50.
[0070]
FIG. 11 shows a process in which the game control unit 32 in the game device 50 performs motion detection based on the image from the controller type imaging device 60 at this time.
[0071]
In step S31, the game control unit 32 determines whether an image has been input. If an image is input, the input compressed image is expanded (decompressed) in step S32.
[0072]
In step S33, the game control unit 32 extracts the identifier 14 from the image information.
[0073]
Subsequently, in step S34, the game control unit 32 stores the coordinates of the extracted identifier with respect to the angle of view, and corrects the direction / angle and the aiming point 16 with respect to the display unit 12 of the controller type imaging device 60, which will be described later. During operation, the position is predicted from the reference relative position of the identifier, and the position of the invisible identifier is predicted. At this time, the above-described position prediction shown in FIG. 7 is performed.
[0074]
In step S <b> 35, the game control unit 32 detects the position of the aiming point 16 from the relative positions of the plurality of identifiers including the identifier supplemented by the prediction.
[0075]
In step S36, the detected position of the aiming point 16 is further corrected by moving object prediction, noise is removed, and the result is reflected in the game input as a control signal.
[0076]
As described above, the position detection system 2 illustrated in FIG. 9 indicates that when the user pulls the trigger of the controller type imaging device 60 as an input device, the image information and the trigger are pulled from the controller type imaging device 60. The trigger information is transmitted to the game apparatus 50. At this time, the game control unit 32 recognizes the identifier 14 from the image information, and analyzes to which position of the monitor 40 the controller type imaging device 60 is directed from the relative positions of the plurality of identifiers. The analyzed result is sent to the game control unit 32 together with the trigger information, converted into a control signal and displayed as the aiming point 16, and used as game input information.
[0077]
Therefore, the position detection system 2 shown as the second embodiment detects the identifier 14 displayed together with the background image on the monitor 40 from the image of the photographing range 17 photographed by the controller type imaging device 60, and the identifier The aiming point 16 can be displayed on the monitor 40 in accordance with the 14 coordinates.
[0078]
As a result, the position detection system 2 uses the movement of the game player (user) and the actual shooting operation as input information in the miniaturized home game device 50, for example, in a virtual world such as a game. It is possible to reflect the operation of. In the position detection system 2, the controller type imaging device 60 can be used not only as an input device of the game apparatus 50 but also as a normal imaging device.
[0079]
The position detection system 2 can also perform motion detection in the game control unit 32 in the imaging control unit 28 of the controller type imaging device 60. The process in the imaging control unit 28 in this case is shown in FIG.
[0080]
The imaging control unit 28 constantly monitors and detects imaging conditions. In step S41, the imaging control unit 28 determines whether the imaging conditions are appropriate.
[0081]
If the imaging control unit 28 determines that the imaging conditions are not appropriate in step S41, the imaging control unit 28 proceeds to step S42, and the imaging driving unit 27 that controls the iris (Iris), zoom (Zoom), and focus (Focus), and the TG unit 26. Then, the AGC unit 23 and the signal processing unit 25 are immediately controlled to adjust the imaging condition.
[0082]
If it is determined in step S41 that the imaging conditions are appropriate, the process proceeds to step S43, where it is determined whether a motion detection request has been issued.
[0083]
If there is no motion detection request in step S43, the process of step S41 is repeated. When it is requested to perform motion detection, in step S44, the imaging control unit 28 acquires an image of the imaging range 17, and extracts the identifier 14 from the image information.
[0084]
In step S45, the imaging control unit 28 stores the coordinates of the extracted identifier with respect to the angle of view, and corrects the direction / angle and the aiming point 16 with respect to the display unit 12 of the controller-type imaging device 60. The position is predicted from the reference relative position of the identifier, and the position of the invisible identifier is predicted. At this time, the position prediction shown in FIG. 7 is performed.
[0085]
In step S46, the imaging control unit 28 detects the position of the aiming point 16 from the relative positions of the plurality of identifiers including the identifier supplemented by the prediction.
[0086]
In step S47, the imaging control unit 28 further corrects the detected position of the aiming point 16 by moving object prediction, removes noise, and reflects it as a control signal in the input of the game.
[0087]
The imaging control unit 28 outputs the detected coordinates of the aiming point 16 to the game control unit 32.
[0088]
As described above, when motion detection is performed in the imaging control unit 28, there is no delay in processing to transfer via the I / F units 51 and 61, compared to when motion detection is performed in the game control unit 32. Therefore, the motion detection operation is further speeded up.
[0089]
In the position detection system 2 shown as the second embodiment, a vector calculation unit for performing vector calculation is provided, and the motion detection similar to the position detection systems 1 and 2 described above may be performed in the vector calculation unit. Good. Such a position detection system is shown in FIG. 13 as a third embodiment.
[0090]
A position detection system 3 shown as the third embodiment includes a vector arithmetic unit 53 in the game apparatus 50. The vector calculation unit 53 is hardware dedicated to vector calculation for performing high-speed image processing unique to a game or the like.
[0091]
In the position detection system 3 shown in FIG. 13, the game control unit 32 includes a main memory 30, a mechanical I / F unit 31, a ROM 34, an image processing unit 35, and a frame memory 36 connected via a bus 37. Although the configuration is the same, the basic configuration of the position detection system 3 shown in FIG. 13 and the position detection system 2 shown in FIG. 9 is the same.
[0092]
In the game system shown in FIG. 13, the vector calculation unit 53 calculates a vector field from the frame captured in the current frame and the image captured in the previous frame stored in the main memory 30. Subsequently, the vector calculation unit 53 extracts feature points from the vector field, and performs motion detection by following the corresponding vector field. The game control unit 32 treats the result of motion detection by the vector arithmetic unit 53 as a control signal and reflects it in the input information.
[0093]
The concept of motion detection using a vector field will be specifically described with reference to FIGS.
[0094]
In the calculation of the vector field, as shown in FIG. 14, first, a change in density is obtained by taking the difference between the current frame and the previous frame on the basis of the density curved surface. Thereafter, the motion vector field for the density curved surface is calculated by multiplying the current frame again by the change in density.
[0095]
Next, an outline of a procedure for calculating a motion vector from a density change will be described with reference to FIG. The motion detection method proposed here is based on correlation using optical flow. First, when the time t transitions from 0 to 1, the density Q is Gaussian distributed with respect to the position X. When t = 0, the center is located at X = 1, and when t = 1, the center is X = Assume a density change that moves to position 2. In this proposal, since the motion is detected from the density transition in the basic cell of X = 1, 2, 3, 3, it is assumed that the density cannot be sampled at an intermediate position, and the motion vector corresponding to the density transition is obtained for each basic cell. Is intended to output. Here, as an example, the motion vector V at the position X = 2 _X Calculate
[0096]
Assuming that the correlation with the movement in the positive direction on the X axis at the position X = 2 at t = 1 is C2,1, the correlation with the movement with respect to the sampling rate X = 1 increments must be maximized. Therefore, the density Q at the position X = 1 when t = 0 _1,0 And the density Q at the position X = 2 when t = 1. _2,1 It is necessary to calculate around the correlation. Therefore, in this proposal, C _2,1 = Q _1,0 × Q _2,1 As a correlation. Similarly, the correlation with respect to the movement in the negative direction on the X axis at the position X = 2 when t = 1 is represented by C _2,3 Then it is C _2,3 = Q _3,0 × Q _2,1 Can be calculated as Finally, the motion vector V at the position X = 2 _X Is the correlation value C _2,1 -C _2,3 Can be calculated as
[0097]
In general, a game apparatus includes an arithmetic processing device specialized for vector arithmetic as hardware. Since such a vector operation unit can increase the vector operation processing speed, motion detection in the calculation of a correlation that can be easily rewritten into the vector operation as described above is effective. As described above, the motion vector is calculated from the density change.
[0098]
Subsequently, processing when the vector calculation unit 53 in the position detection system 3 detects motion at this time will be described with reference to FIG. 16, and motion detection is executed by a series of processing shown in FIG. It is not limited.
[0099]
In step S51, the game control unit 32 in the game device 50 determines whether or not a captured image has been input. The game control unit 32 repeatedly performs the process in step S51 until a captured image is input.
[0100]
When the captured image is input, the game control unit 32 calculates a density curved surface of the pixel value using the vector calculation unit 53 in step S52.
[0101]
In the subsequent step S53, the game control unit 32 detects and corrects the movement of the entire image and performs statistical processing to extract and correct the blur.
[0102]
In step S54, the vector calculation unit 53 divides the corrected image into grid blocks, and calculates a vector field by extracting a motion vector for each block.
[0103]
In step S55, the game control unit 32 extracts feature objects by collecting consecutive similar vectors.
[0104]
Subsequently, in step S56, it is recognized that the characteristic object is the identifier 14 by performing pattern matching and tracking determination.
[0105]
In step S57, when there is an invisible identifier, the game control unit 32 predicts the position of the invisible identifier from the reference relative positions of the plurality of visible identifiers, and detects the position of the aiming point 16 from the relative positions of these identifiers. .
[0106]
In step S58, the game control unit 32 converts the detected position of the aiming point 16 into a control signal and reflects it as input information to the game.
[0107]
As described above, according to the position detection system 3 provided with the vector calculation unit 53, the feature point extraction process that relies on the color, the pattern, etc., can perform motion detection with higher accuracy and improved real-time control. .
[0108]
Here, a function of performing imaging at an arbitrary timing by the above-described asynchronous shutter and a function of performing imaging by multiple exposure shooting in which a plurality of images are captured within one cycle of a synchronization signal for imaging will be described. The following is a description of Japanese Patent Application Laid-Open No. 9-247556 (name of the invention “imaging device provided with an individual imaging unit”) used by the present invention as a prior art. In this prior art, the invention is applied to an imaging device.
[0109]
For example, as shown in FIG. 17, the imaging device includes a plurality of light receiving element portions that perform photoelectric conversion and are arranged in a number of parallel rows, and a CCD that transfers charges obtained in each light receiving element portion. A solid-state imaging unit 111 having a light receiving / charge transfer unit provided with the formed charge transfer region is provided. Further, an optical system including a diaphragm mechanism 112, a lens system 113, and the like is disposed in front of the solid-state imaging unit 111. This optical system is a light receiving / charge transfer unit included in the solid-state imaging unit 111. A subject image is projected by allowing external light to enter.
[0110]
As shown in FIG. 18, the light receiving / charge transferring section includes a plurality of light receiving element sections 116, each of which constitutes an individual pixel, on a semiconductor substrate 115, extending in a number of horizontal directions (in the direction of the arrow h). (Pixel horizontal row) is formed and arranged.
The light receiving element portion 116 forming each of a large number of pixel horizontal columns also forms a large number of parallel columns (pixel vertical columns) extending in the vertical direction (the direction of the arrow v). A vertical charge transfer unit 117 formed by a CCD group is arranged along each pixel vertical column formed by 116. Each vertical charge transfer unit 117 is driven by, for example, two-phase vertical transfer drive signals φV1 and φV2 to perform a charge transfer operation. A charge readout gate unit 118 is provided between each of the plurality of light receiving element units 116 forming each pixel vertical column and the vertical charge transfer unit 117 corresponding to the vertical column.
[0111]
Further, as shown in FIG. 19, a channel stopper 119 and an overflow control unit 120 are formed around each light receiving element unit 116. Further, a drain part 121 is arranged adjacent to the overflow control part 120, and the channel stopper part 122 distinguishes between the drain part 121 and the adjacent vertical charge transfer part 117.
[0112]
An insulating layer is disposed on each of the above-described portions, and the vertical transfer electrodes E1 and E2 extending in the horizontal direction are alternately disposed in the vertical direction on the vertical charge transfer portion 117 via the insulating layer. The vertical transfer electrode E1 is configured by the storage unit electrode E1c and the transfer unit electrode E1t, and the vertical transfer electrode E2 is configured by the storage unit electrode E2c and the transfer unit electrode E2t. The vertical transfer drive signals φV1 and φV2 are supplied to the vertical transfer electrodes E1 and E2, respectively.
[0113]
A read gate electrode EG extending in the vertical direction is disposed on the charge read gate portion 118, and an overflow gate electrode ED extending in the vertical direction is disposed on the overflow control portion 120. A read gate control signal SG and an overflow gate control signal SD are supplied to the read gate electrode EG and the overflow gate electrode ED, respectively. Then, a light shielding layer is disposed on the other portions except the light receiving element portion 116.
[0114]
One end side of each of the plurality of vertical charge transfer units 117 is connected to a horizontal charge transfer unit 123 formed by a CCD group and extending in the horizontal direction at the edge of the semiconductor substrate 115. The horizontal charge transfer unit 123 is driven by, for example, two-phase horizontal transfer drive signals φH1 and φH2 to perform a charge transfer operation. An output unit 24 is provided at one end of the horizontal charge transfer unit 123, and an output terminal 125 is led out from the output unit 124.
[0115]
In addition, on the other end side of each of the plurality of vertical charge transfer units 117 in the semiconductor substrate 115, a charge absorption unit 126 extending in the horizontal direction is provided.
[0116]
When a still image imaging operation is performed in the solid-state imaging unit 111 having the light receiving / charge transfer unit illustrated in FIGS. 18 and 19, a predetermined light receiving period for the light receiving / charge transfer unit is set, During the light receiving period, each of a large number of light receiving element units 116 receiving light from an imaging target incident on the light receiving / charge transfer unit through an optical system including the lens system 113, the diaphragm mechanism 112, etc. performs photoelectric conversion. Charges corresponding to the received light are accumulated. Thereafter, the plurality of charge read gate units 118 transfer the charges accumulated in the corresponding light receiving element units 116 in accordance with the read gate control signal SG from the drive signal forming unit 130 supplied thereto, and the corresponding vertical charge transfer units 117. Read out.
[0117]
Subsequently, the charges read to each vertical charge transfer unit 117 are supplied to the vertical transfer electrodes E1 and E2 in the solid-state imaging unit 111 as the signal charge transfer drive signal ST from the drive signal forming unit 130, respectively. The horizontal charge transfer is sequentially performed as signal charges for the signals obtained by the plurality of light receiving element portions 116 forming each pixel horizontal column by the charge transfer operation of each vertical charge transfer portion 117 driven by the transfer drive signals φV1 and φV2. The data is transferred toward the unit 123. The horizontal charge transfer unit 123 transfers the signal charge as a signal charge by a charge transfer operation performed by being driven by the two-phase horizontal transfer drive signals φH1 and φH2 supplied as the signal charge transfer drive signal ST from the drive signal forming unit 130. Charges obtained by the plurality of light receiving element portions 116 forming one pixel horizontal row are supplied to the output unit 124. In the output unit 124, the charges transferred by the horizontal charge transfer unit 123 are sequentially signaled and led to the output terminal 125, and imaging based on the charges accumulated in the plurality of light receiving element units 116 at the output terminal 125. An imaging output signal IP corresponding to the object is obtained.
[0118]
In such a case, horizontal charge transfer by each vertical charge transfer unit 117 addressed to one pixel horizontal column with respect to the charges obtained by the plurality of light receiving element units 116 forming each of all the pixel horizontal columns in the light receiving / charge transfer unit. The transfer to the unit 123 is completed within each frame period, and the horizontal charge of the charges obtained by the plurality of light receiving element units 116 forming the one-pixel horizontal row transferred to the horizontal charge transfer unit 123 is horizontal. The supply to the output unit 124 by the charge transfer unit 123 is completed within each line period, and the two-phase vertical transfer drive signals φV1 and φV2 and the two-phase signals that are the signal charge transfer drive signal ST are used. Each of the horizontal transfer drive signals φH1 and φH2 is set.
Therefore, the imaging output signal IP derived to the output terminal 125 is repeated for a frame period formed by a unit of line periods.
[0119]
The drive signal forming unit 130 is supplied with the charge read timing signal TG, the signal charge transfer timing signal TT, the charge sweep transfer timing signal TS, and the charge discharge timing signal TD from the timing signal generating unit 131, and according to the charge read timing signal TG. The read gate control signal SG formed in this manner is sent, and the two-phase vertical transfer drive signals φV1 and φV2 and the two-phase horizontal transfer drive signals φH1 and φH2 formed in response to the signal charge transfer timing signal TT are signal-charge transferred. State to send as signal ST and State to send two-phase vertical transfer drive signals φV2 and φV1 and two-phase horizontal transfer drive signals φH2 and φH1 formed according to charge sweep transfer timing signal TS as sweep transfer drive signal SS Is selected according to the charge discharge timing signal TD. The generated overflow gate control signal SD is sent out.
[0120]
The timing signal generation unit 131 is supplied with the frame synchronization signal SF and the line synchronization signal SH from the synchronization signal generation unit 132, and further receives the light reception period signal SE from the control unit 133 forming the operation control unit, and the asynchronous normal light reception mode setting. A signal SN, an asynchronous shutter standby mode setting signal SWS, a synchronous return command signal SR, and a multiple exposure mode setting signal SMM are also supplied.
[0121]
In the timing signal generator 131, the light reception period signal SE from the control unit 133, the asynchronous normal light reception mode setting signal SN, the asynchronous shutter standby mode setting signal SWS, the synchronous return command signal SR, and the multiple exposure mode setting signal SMM. Depending on the state of the frame synchronization signal SF and the line synchronization signal SH, that is, between the frame synchronization signal SF and the line synchronization signal SH in the same phase or the frame synchronization signal SF and the line synchronization signal, respectively. The internal frame synchronization signal SFO and the internal line synchronization signal SHO, each having a certain phase difference from the SH, are generated asynchronously with the frame synchronization signal SF and the line synchronization signal SH. Internal frame synchronization signal SFO and internal line synchronization signal SHO A condition that occurs is taken selectively.
[0122]
Further, the drive signal forming unit 130 obtains the internal frame synchronization signal SFO and the internal line synchronization signal SHO that are synchronized with the frame synchronization signal SF and the line synchronization signal SH, respectively, in the timing signal generation unit 131. A charge read timing signal TG, a signal charge transfer timing signal TT, a charge sweep transfer timing signal TS, and a charge discharge timing signal TD, which are formed based on the internal frame synchronization signal SFO and the internal line synchronization signal SHO, are supplied to read the charge. An operation state in which the read gate control signal SG formed according to the timing signal TG is sent, and the internal frame synchronization signal SFO and the internal state which are made asynchronous with respect to the frame synchronization signal SF and the line synchronization signal SH in the timing signal generator 131 Line synchronization signal SH Are obtained based on the internal frame synchronization signal SFO and the internal line synchronization signal SHO, the charge read timing signal TG, the signal charge transfer timing signal TT, the charge sweep transfer timing signal TS, and the charge discharge timing signal. TD is supplied, and an operation state in which a read gate control signal SG formed in response to the charge read timing signal TG is selectively taken.
[0123]
The frame synchronization signal SF and the line synchronization signal SH from the synchronization signal generator 132, and the internal frame synchronization signal SFO and the internal line synchronization signal SHO formed in the timing signal generator 131 are also supplied to the control unit 133. The charge read timing signal TG, the signal charge transfer timing signal TT, the charge sweep transfer timing signal TS and the charge discharge timing signal TD obtained in the timing signal generator 131 are also supplied to the control unit 133. Further, the control unit 133 forming the operation control unit includes an operation mode designation signal SM through the terminal 134, a shutter speed designation signal SSV through the terminal 135, and a shutter operation which is an operation for causing the apparatus to perform an imaging operation. A shutter signal SSH that is a light reception command signal that is generated in response to the signal is supplied through a terminal 136.
[0124]
The imaging output signal IP obtained at the output terminal 125 in the light receiving / charge transfer unit of the solid-state imaging unit 111 shown in FIG. 18 is amplified and sampled by the automatic gain control (AGC) amplification unit 140 as shown in FIG. -It is supplied to the hold unit 141. The sampling and holding unit 141 performs level sampling and sample level holding for each predetermined short period for the imaging output signal IP to obtain a sampling and holding output signal SI, which is analog / digital (A / D) Supplied to the conversion unit 142. The A / D conversion unit 142 digitizes the imaging output signal IP based on the sampling and holding output signal SI, and the A / D conversion unit 142 generates a digital imaging signal DI corresponding to the imaging output signal IP. Obtained and supplied to the imaging signal digital processing unit 143.
[0125]
In the controller type imaging device 11 of the position detection system 1 shown in FIG. 2, the signal output from the S / H unit 22 is processed in the AGC unit 23, which is different from the signal processing procedure of the imaging device described above. However, the controller-type imaging device 11 of the position detection system 1 may process the signal processed in the AGC amplification unit 140 in the sample hold unit (S / H unit) 141 as in the above-described imaging device. Good.
[0126]
The control signal DCC and the control data DCD are exchanged between the imaging signal digital processing unit 143 and the control unit 133, and the imaging signal digital processing unit 143 responds to the exchange of the control data DCC and the control data DCD. Thus, various digital processes are performed on the imaging output signal IP. As a result, the digital imaging output signal DIO is derived from the imaging signal digital processing unit 143 to the output terminal 144.
[0127]
Under such circumstances, when a still image capturing operation is performed, an operation mode designation signal SM for designating an operation mode to be selected is supplied to the control unit 133 through the terminal 134 and should be selected. A shutter speed designation signal SSV for designating the shutter speed is supplied. The operation mode designated by the operation mode designation signal SM is, for example, an asynchronous normal light receiving mode, an asynchronous shutter standby light receiving mode, a multiple exposure mode, or the like. Hereinafter, processing of the imaging apparatus in each mode will be described.
[0128]
When the operation mode designated by the operation mode designation signal SM is the asynchronous normal light reception mode, the control unit 33 supplies the asynchronous normal light reception mode setting signal SN to the timing signal generator 131. In response to the asynchronous normal light reception mode setting signal SN, the timing signal generator 131 receives the frame synchronization signal SF from the synchronization signal generator 132 shown in FIG. 20A as seen in the period before time ta in FIG. A timing signal SXO as shown in E of FIG. 20 is formed based on the internal frame synchronization signal SFO shown in D of FIG. 20 having a certain phase difference between and.
[0129]
At this time, the timing signal generator 131 sends out a charge read timing signal TG, a signal charge transfer timing signal TT, a charge sweep transfer timing signal TS and a charge discharge timing signal TD corresponding to the internal frame synchronization signal SFO. As shown in the period before time ta in FIG. 20, the read gate control signal SG as shown in F of FIG. 20 and the G of FIG. 20 are synchronized with the internal frame synchronization signal SFO as seen from the drive signal forming unit 130. A sweep transfer drive signal SS as shown in FIG. In the period before time ta in FIG. 20, as shown in J of FIG. 20, the SF synchronization mode synchronized with the frame synchronization signal SF is taken.
[0130]
Under such circumstances, when a shutter operation is performed at time ta in FIG. 20, a shutter signal SSH as a light reception command signal as shown in FIG. 20B is supplied to the control unit 133 through the terminal 136.
[0131]
The control unit 133 supplies a light reception period signal SE as shown in FIG. 20C to the timing signal generator 131 in accordance with the leading edge of the shutter signal SSH. At this time, as shown in C and D of FIG. 20, the timing signal generator 131 resets the formation of the internal frame synchronization signal SFO by the leading edge of the light receiving period signal SE, and uses the internal frame synchronization signal SFO as the frame synchronization signal. Asynchronous with SF, the formation of the timing signal SXO is interrupted, and the charge read timing signal TG and the charge discharge timing signal TD corresponding to the leading edge of the light receiving period signal SE are sent out.
[0132]
Then, as shown in F and G of FIG. 20, the read gate control signal SG and the sweep transfer drive signal SS corresponding to the leading edge of the light reception period signal SE are received from the drive signal forming unit 130 by the solid-state imaging unit 111.・ Sent to the charge transfer unit. Thereby, in the light receiving / charge transfer unit of the solid-state imaging unit 111, each charge readout gate unit 118 is opened according to the readout gate control signal SG, and the charge in each light receiving element unit 116 is transferred to each vertical charge transfer unit 117. The vertical charge transfer unit 117 reads the charge read through the open charge read gate unit 118 in accordance with the sweep transfer drive signal SS, and reads the charge read to the horizontal charge transfer unit in the semiconductor substrate 115. A charge sweeping operation for transferring to the charge absorbing portion 126 provided on the opposite side to the 123 is performed. The charge transferred by the charge sweeping is absorbed by the charge absorbing unit 126. The charge sweep period during which such charge sweep operation is performed is, for example, about 27 line periods.
[0133]
Then, at the leading edge of the light reception period signal SE, a light reception period for the light reception / charge transfer section of the solid-state imaging unit 111 including the charge sweeping period is started. As shown in J of FIG. It will shift to the running mode. In the light reception period, as shown in FIG. 20D, the internal frame synchronization signal SFO which is reset at the leading edge of the light reception period signal SE and is asynchronous with the frame synchronization signal SF is obtained. SXO is not obtained. As a result, in the light receiving period, the readout gate control signal SG is not obtained after being obtained according to the leading edge of the light receiving period signal SE, and the charge readout gate portion except for the short period at the beginning of the charge sweeping period. 118 is maintained in a closed state, and charges are accumulated in a large number of light receiving element portions 116 by light reception as shown in I of FIG.
[0134]
FIG. 21 shows an enlarged detailed timing relationship between the shutter signal SSH supplied to the control unit 133, the internal line synchronization signal SHO, the light receiving period signal SE, the readout gate control signal SG, and the sweep transfer drive signal SS. In the timing relationship shown in FIG. 21, when the leading edge of the shutter signal SSH arrives at time ta, the leading edge of the light receiving period signal SE is obtained at the time of the leading edge of the first internal line synchronization signal SHO thereafter. It is done. Then, at the time of the leading edge of the light receiving period signal SE, the leading edge of the read gate control signal SG and the leading edge of the sweep transfer drive signal SS are obtained, and the mode shifts from the SF synchronous mode to the free-running mode. In other words, the leading edge of the light receiving period signal SE is slightly delayed (at most about one line period) from the leading edge of the shutter signal SSH.
[0135]
After that, as shown in FIG. 20, when the trailing edge of the light receiving period signal SE supplied from the control unit 133 to the timing signal generating unit 131 arrives at the time tb, the timing signal generating unit 131 receives the D in FIG. As indicated by E, the internal frame synchronization signal SFO is obtained according to the trailing edge of the light receiving period signal SE, and the timing signal SXO is obtained according to the leading edge of the internal frame synchronization signal SFO. The time tb at the trailing edge of the light receiving period signal SE is set by the control unit 133 according to the shutter speed designated by the shutter speed designation signal SSV supplied thereto. Then, the timing signal generator 131 sends out a charge read timing signal TG and a signal charge transfer timing signal TT in response to the timing signal SXO. Accordingly, as shown in F and H of FIG. 20, the drive signal forming unit 130 responds to the read gate control signal SG having the leading edge corresponding to the trailing edge of the light receiving period signal SE and the trailing edge of the light receiving period signal SE. The signal charge transfer drive signal ST having the leading edge is sent to the light receiving / charge transferring unit of the solid-state imaging unit 111, and the light receiving period ends.
[0136]
As a result, in the light receiving / charge transfer unit of the solid-state imaging unit 111, each charge readout gate unit 118 is opened according to the readout gate control signal SG, and as shown in I of FIG. In addition to reading out the charges in the unit 116 to each vertical charge transfer unit 117, each vertical charge transfer unit 117 is read out through the charge read gate unit 118 which is opened in accordance with the signal charge transfer drive signal ST. A charge transfer operation is performed to transfer the generated charges as signal charges to the horizontal charge transfer unit 123 in the semiconductor substrate 115. Further, a charge transfer operation is performed in which the horizontal charge transfer unit 123 transfers the charges transferred by the vertical charge transfer units 117 to the output unit 124 as signal charges in accordance with the signal charge transfer drive signal ST. . The charge transferred to the output unit 124 is converted into an imaging output signal IP by the output unit 124, and the imaging output signal IP is led to the output terminal 125.
[0137]
In the charge transfer period in which the charge transfer operation by each of the vertical charge transfer units 117 and the horizontal charge transfer unit 123 is performed, the timing signal generation unit 31 thereafter performs internal frame synchronization as shown in FIGS. At the time tc when the signal SFO is obtained and the timing signal SXO is obtained according to the leading edge of the internal frame synchronization signal SFO, the trailing edge of the signal charge transfer drive signal ST arrives and ends. Then, the timing signal generation unit 131 sends out the charge read timing signal TG and the signal charge transfer timing signal TT in response to the timing signal SXO, thereby causing the drive signal forming unit 130 to show the signals F and G in FIG. As shown, the readout gate control signal SG having a leading edge corresponding to the trailing edge of the signal charge transfer driving signal ST and the sweeping transfer driving signal SS having a leading edge corresponding to the trailing edge of the signal charge transfer driving signal ST are solid-state imaging. The charge is discharged to the light receiving / charge transfer unit of the unit 111 to perform the charge sweeping operation.
[0138]
After that, at time td, as shown in FIG. 20D, the internal frame synchronization signal SFO obtained by the timing signal generator 131 again has a certain phase difference from the frame synchronization signal SF. Thus, as shown in FIG. 20J, the self-running mode returns to the SF synchronous mode.
[0139]
As described above, in the case where the still image capturing operation is performed under the asynchronous normal light receiving mode, when the shutter operation is performed, a maximum delay of approximately one line period from the time of the shutter operation is delayed. At the time, the light receiving period for the light receiving / charge transfer unit of the solid-state imaging unit 111 is started, and the charge sweeping operation ends from the start. For example, after the 27th line period, a large number of light receiving element units in the light receiving / charge transfer unit A light receiving state in which effective charge accumulation is performed in 116 is obtained. Therefore, the quickness of the light reception response operation when the shutter operation is performed is greatly improved.
[0140]
Next, the asynchronous shutter standby light receiving mode will be described. When the operation mode designated by the operation mode designation signal SM is the asynchronous shutter standby light receiving mode, the control unit 133 supplies the asynchronous shutter standby mode setting signal SWS to the timing signal generator 131. In response to the asynchronous shutter standby mode setting signal SWS, the timing signal generator 131 receives the frame synchronization signal SF from the synchronization signal generator 132 shown in FIG. 22A as seen in the period before the time te in FIG. 22, the internal frame synchronization signal SFO shown in D of FIG. 22 having a certain phase difference is obtained, and after the time te, the formation of the timing signal SXO is stopped and after the time te In accordance with the leading edge of the first internal frame synchronization signal SFO, continuous transmission of the charge read timing signal TG and the charge sweep transfer timing signal TS is started. Thereby, as shown in FIG. 22B, the transition from the normal light receiving state to the shutter waiting state is performed at time te.
[0141]
With the start of continuous transmission of the charge read timing signal TG and the charge sweep transfer timing signal TS from the timing signal generator 131, as shown in G and H of FIG. In accordance with the leading edge of the internal frame synchronization signal SFO, the read gate control signal SG and the sweep transfer drive signal SS are continuously sent to the light receiving / charge transfer unit of the solid-state imaging unit 111. Thereby, in the light receiving / charge transfer unit of the solid-state imaging unit 111, each charge reading gate unit 118 is continuously opened according to the reading gate control signal SG, and the charge in each light receiving element unit 116 is changed to each vertical charge. The charge reading to the transfer unit 117 is continuously performed, and the charges read by the vertical charge transfer units 117 through the charge reading gate unit 118 that is continuously opened in response to the sweep transfer drive signal SS. Then, a charge sweeping operation for continuously transferring to the charge absorbing portion 126 provided on the opposite side to the horizontal charge transferring portion 123 in the semiconductor substrate 115 is performed. The charge transferred by the charge sweeping is absorbed by the charge absorbing unit 126. In the charge sweeping period in which such a continuous charge sweeping operation is performed, as shown in FIG. 22J, no charge is accumulated in the light receiving / charge transfer unit of the solid-state imaging unit 111.
[0142]
When the shutter operation is performed at time tf in FIG. 22 while the charge sweeping operation is continuously performed in the light receiving / charge transfer unit of the solid-state imaging unit 111, as shown in FIG. 22C. The shutter signal SSH is supplied to the control unit 133 through the terminal 136. The control unit 133 supplies a light reception period signal SE as shown in F of FIG. 22 to the timing signal generator 131 in accordance with the leading edge of the shutter signal SSH. At this time, as shown in FIGS. 22D and 22E, the timing signal generator 131 resets the formation of the internal frame synchronization signal SFO by the leading edge of the light reception period signal SE, and uses the internal frame synchronization signal SFO as the frame synchronization signal. Asynchronous with the SF, the formation of the timing signal SXO is interrupted, and the continuous transmission of the charge read timing signal TG and the charge sweep transfer timing signal TS is terminated.
[0143]
As a result, the drive signal forming unit 130 stops the continuous transmission of the read gate control signal SG and the sweep transfer drive signal SS in accordance with the leading edge of the light receiving period signal SE as shown in G and H of FIG. Then, as shown in FIG. 22F, at the leading edge of the light reception period signal SE, the light reception period for the light reception / charge transfer section of the solid-state imaging unit 111 is started without including the charge sweeping period. As shown in B, a transition from the shutter standby state to the normal light receiving state is performed.
[0144]
In the light reception period, as shown in FIG. 22D, the internal frame synchronization signal SFO which is reset at the leading edge of the light reception period signal SE and is asynchronous with the frame synchronization signal SF is obtained. SXO is not obtained. As a result, during the light receiving period, the readout gate control signal SG is not obtained, the charge readout gate unit 118 is maintained in the closed state, and charge accumulation by light reception in a large number of light receiving element units 116 is shown in FIG. It is done as you can.
[0145]
After that, when the trailing edge of the light reception period signal SE supplied from the control unit 133 to the timing signal generator 131 arrives at the time point tg, the timing signal generator 131 receives the light reception period as shown in D and E of FIG. An internal frame synchronization signal SFO is obtained according to the trailing edge of the signal SE, and a timing signal SXO is obtained according to the leading edge of the internal frame synchronization signal SFO. The time tf at the trailing edge of the light receiving period signal SE is set by the control unit 133 according to the shutter speed designated by the shutter speed designation signal SSV supplied thereto. Then, the timing signal generator 131 sends out a charge read timing signal TG and a signal charge transfer timing signal TT in response to the timing signal SXO. As a result, as shown in G and I of FIG. 22, the drive signal forming unit 130 responds to the read gate control signal SG having the leading edge corresponding to the trailing edge of the light receiving period signal SE and the trailing edge of the light receiving period signal SE. The signal charge transfer drive signal ST having the leading edge is sent to the light receiving / charge transferring unit of the solid-state imaging unit 111, and the light receiving period ends.
[0146]
As a result, in the light receiving / charge transfer unit of the solid-state imaging unit 111, each charge readout gate unit 118 is opened according to the readout gate control signal SG, and as shown in FIG. In addition to reading out the charges in the unit 116 to each vertical charge transfer unit 117, each vertical charge transfer unit 117 is read out through the charge read gate unit 118 which is opened in accordance with the signal charge transfer drive signal ST. A charge transfer operation is performed to transfer the generated charges as signal charges to the horizontal charge transfer unit 123 in the semiconductor substrate 115. Further, a charge transfer operation is performed in which the horizontal charge transfer unit 123 transfers the charges transferred by the vertical charge transfer units 117 to the output unit 124 as signal charges in accordance with the signal charge transfer drive signal ST. . The charge transferred to the output unit 124 is converted into an imaging output signal IP by the output unit 124, and the imaging output signal IP is led to the output terminal 125.
[0147]
During the charge transfer period in which the charge transfer operation by each vertical charge transfer unit 117 and horizontal charge transfer unit 123 is performed, the timing signal generation unit 131 thereafter performs internal frame synchronization as shown in D and E of FIG. At the time point th when the signal SFO is obtained and the timing signal SXO is obtained in accordance with the leading edge of the internal frame synchronization signal SFO, the trailing edge of the signal charge transfer drive signal ST arrives and ends. Then, the timing signal generator 131 sends out the charge read timing signal TG and the charge sweep transfer timing signal TS in accordance with the timing signal SXO, thereby causing the drive signal forming unit 130 to show the signals G and H in FIG. As shown, the read gate control signal SG having a leading edge corresponding to the trailing edge of the signal charge transfer driving signal ST and the sweeping charge transfer driving signal SS having a leading edge corresponding to the trailing edge of the signal charge transfer driving signal ST are solidified. The charge is discharged to the light receiving / charge transfer unit of the imaging unit 111 and the charge sweeping operation is performed.
[0148]
Thereafter, at time ti, the control unit 133 sends a synchronization return command signal SR to the timing signal generator 31, and thereby, as shown in FIG. 22D, the internal frame synchronization signal obtained in the timing signal generator 131. The SFO again has a certain phase difference from the frame synchronization signal SF, and the SF synchronization is restored.
[0149]
As described above, when the still image capturing operation is performed under the asynchronous shutter standby light receiving mode, the shutter standby state is taken in advance, and the charge sweeping operation is continuously performed in the light receiving / charge transfer unit of the solid-state imaging unit 111. When the shutter operation is performed, the continuous charge sweeping operation performed until the time of the shutter operation is terminated, and immediately, the light receiving / charge transfer unit of the solid-state imaging unit 111 As a result, a light receiving state in which effective charge accumulation is performed in a large number of light receiving element portions 116 in the light receiving / charge transferring portion is obtained. Therefore, when a shutter operation is performed, an extremely quick light reception response operation is performed.
[0150]
Next, the multiple exposure mode will be described. When the operation mode designated by the operation mode designation signal SM is the multiple exposure mode, the control unit 133 supplies the multiple exposure mode setting signal SMM to the timing signal generator 131. In response to the multiple exposure mode setting signal SMM, the timing signal generator 131 generates an internal frame synchronization signal SFO for the frame synchronization signal SF from the synchronization signal generator 132 shown in FIG. As shown in C, three periods of the internal frame synchronization signal SFO are set as unit periods, and a predetermined number, for example, seven timing signals SXO, for example, seven timing signals SXO are generated with a constant period within each unit period.
[0151]
In addition, the timing signal generator 131 sends out a charge read timing signal TG corresponding to the leading edge of each of a predetermined number of timing signals SXO in each unit period of three cycles of the internal frame synchronization signal SFO, The charge discharge timing signal TD is continuously transmitted during the period from the trailing edge of the last number of timing signals SXO to the end of the unit period, and each of the three periods of the internal frame synchronization signal SFO The signal charge transfer timing signal TT is sent during a period of one cycle for the final internal frame synchronization signal SFO in the unit period.
[0152]
As a result, the drive signal forming unit 130, in accordance with the charge readout timing signal TG, as shown in FIG. 23D, a predetermined number within each unit period for three cycles of the internal frame synchronization signal SFO, For example, the seven readout gate control signals SG are sent to the light receiving / charge transfer unit of the solid-state imaging unit 111, and the internal frame synchronization signal as shown in FIG. 23E according to the charge discharge timing signal TD. Solid-state imaging of the overflow gate control signal SD that continues during the period from the trailing edge of the predetermined number of read gate control signals SG in each unit period of SFO for the three periods to the end of the unit period To the light reception / charge transfer unit of the unit 111. Further, the drive signal forming unit 130 determines the final internal frame in each unit period corresponding to three cycles of the internal frame synchronization signal SFO, as shown in G of FIG. 23, according to the signal charge transfer timing signal TT. The signal charge transfer drive signal ST is sent to the light receiving / charge transfer unit of the solid-state imaging unit 111 during a period of one cycle of the synchronization signal SFO.
[0153]
As a result, in the light receiving / charge transfer unit of the solid-state imaging unit 111, a predetermined number of, for example, seven read gate control signals SG between each other during each unit period of three cycles of the internal frame synchronization signal SFO. Is a divided light receiving period for the light receiving / charge transfer unit, and as shown in FIG. 23F, in each divided light receiving period, a large number of light receiving element units 116 in the light receiving / charge transfer unit receive charges due to light reception. Is accumulated. Then, the charges obtained in each divided light receiving period are sequentially read out to each vertical charge transfer section 117 by each charge reading gate section 18 which is opened by supplying a reading gate control signal SG in a period following that period. It is.
[0154]
In this way, during each unit period of three periods for the internal frame synchronization signal SFO, each vertical charge transfer unit 117 from each of the light receiving element units 116 in the light receiving / charge transfer unit through each charge read gate unit 118. The charges that are intermittently read out are accumulated and accumulated in each vertical charge transfer unit 117 until the period of one cycle for the final internal frame synchronization signal SFO in each unit period arrives.
As a result, multiple exposure is performed on a large number of light receiving element portions 116 in the light receiving / charge transfer portion during each unit period substantially corresponding to three cycles of the internal frame synchronization signal SFO. .
[0155]
Then, when a period of one cycle for the final internal frame synchronization signal SFO in each unit period arrives, the charges accumulated and accumulated in each vertical charge transfer unit 117 are transferred to the light receiving / charge transfer unit of the solid-state imaging unit 111. By supplying the signal charge transfer drive signal ST, signal charges are transferred from each vertical charge transfer unit 117 to the horizontal charge transfer unit 123 and further from the horizontal charge transfer unit 123 to the output unit 124.
Then, the charge transferred to the output unit 124 is converted into the imaging output signal IP by the output unit 124, and the imaging output signal IP is derived to the output terminal 125.
[0156]
Thus, during the period in which the charges accumulated and accumulated in each vertical charge transfer unit 117 are transferred as signal charges, the charges obtained by receiving light in the many light receiving element units 116 in the light receiving / charge transferring unit of the solid-state imaging unit 111 are: At that time, the overflow gate control signal SD is continuously supplied to the overflow gate electrode ED provided in the light receiving / charge transfer unit, and is discharged to the drain unit 121 through the overflow control unit 120 which is opened. Therefore, at the start of the divided light-receiving period for the light-receiving / charge transfer unit in each unit period, unnecessary charges due to light reception prior to the divided light-receiving period are received in the many light-receiving element units 116 in the light-receiving / charge transfer unit. There is no situation that would be accumulated.
[0157]
In this way, when a still image capturing operation is performed under the multiple exposure mode, substantially three cycles of the internal frame synchronization signal SFO are obtained based on a relatively simple configuration and operation control. During each unit period, the imaging output signal IP based on the signal charges obtained by performing multiple exposure on the large number of light receiving element portions 116 in the light receiving / charge transferring portion is obtained.
[0158]
Consider a case where such multiple exposure photography and asynchronous shutter based on the prior art are applied to the present invention.
[0159]
The number of images corresponding to the number of times of multiple exposure (N) is displayed superimposed on the image that has been shot with multiple exposure. For example, in the case of an image that has been subjected to multiple exposure shooting with N = 2, an image 201 and an image 202 as shown in FIG. Become.
[0160]
In this case, the image 203 displays a plurality of identifiers whose positions change with time. Here, since the case where the number of multiple exposures is N = 2 is considered, it is possible to detect the position of the aiming point twice as accurately in consideration of time series.
The temporal relationship between a plurality of identifiers displayed in one image is inferred from the fact that the stored position changes of past aiming points are continuous.
[0161]
Here, if N = 2 multiple exposure shooting is performed by using a TG (Timing Generator) having a multiple exposure function corresponding to vector calculation processing, a sampling rate equivalent to the conventional one can be expected. Furthermore, if N = 4, it is possible to expect a sampling rate twice that of the prior art, and position detection can be realized at a finer timing. Further, since the transfer rate of the image at this time may be 1 / N of the sampling rate, when applied to the position detection system according to the present invention, the transfer rate can be earned, and a more efficient position detection system. Can be built.
[0162]
For example, USB Ver. Even if transfer is performed at a high-speed mode of 12 MHz conforming to 1.1, it is limited to transfer an image of 30 fps with OVGA 16-bit color number. By performing multiple exposure with N = 4, for example, as shown in FIG. Until the step of extracting the object by collecting successive similar vectors in step S25, more accurate position detection can be realized with a CPU rate that is the same as that at 30 fps and a sampling rate corresponding to 120 fps.
[0163]
Next, consider a case where imaging is performed at an arbitrary timing by an asynchronous shutter. FIG. 25A shows a case where an image is taken at a conventional fixed timing, and FIG. 25B shows a case where a game device to which the present invention is applied takes an image at an arbitrary timing.
[0164]
Conventionally, as shown in FIG. 25 (A), the previous image 210 and the subsequent image 211 are imaged at a fixed timing at an interval of 1/30 seconds, and an arbitrary period from the first imaging to 1/30 seconds. It was impossible to take an image at this timing.
[0165]
By using the asynchronous imaging technique described with reference to FIGS. 20 and 22 in the position detection system according to the present invention, an image 212 can be obtained at an arbitrary timing as shown in FIG. .
[0166]
Note that an image immediately before an image obtained at an arbitrary timing (an image captured at an original fixed timing) 213 cannot be read because it is swept away. However, the image 214 after the image 212 obtained at an arbitrary timing is read in accordance with the CCD reading timing of 1/30 seconds as a normal imaging timing.
[0167]
When a conventional imaging device is used, the position of a feature point can be detected only from an image obtained at a CCD readout timing such as 1/30 second. In addition, as described above, even when the image is taken by multiple exposure shooting, the position can be detected only from the image obtained by time-dividing the CCD readout timing such as 1/30 second by a small integer up to about N = 4. There wasn't. On the other hand, since the position detection system according to the present invention can capture an image at an arbitrary timing and measure the position of a feature point at an arbitrary timing, it is more accurate than the case of estimating the position of a feature point by interpolation. Motion detection is possible.
[0168]
As described above, the position detection system according to the present invention can perform the following position detection as the fourth embodiment and the fifth embodiment by applying the multiple exposure shooting and the shooting by the asynchronous shutter. To do.
[0169]
The position detection system 4 shown as the fourth embodiment of the present invention displays an identifier on a display device, captures a display image including the identifier with an imaging device as an operation input device, and analyzes the captured image in an analysis unit. A position detection system that detects a positional relationship between a display device and an imaging device and an orientation of the imaging device with respect to the display device by non-contact by performing analysis, for example, a predetermined position or a predetermined position of an image displayed on the display device In addition to making it possible to input an image by selecting an image pickup device as an operation input device to face the image, it is possible to perform the selection operation continuously by multiple exposure shooting. This is the user interface.
[0170]
As shown in FIG. 26, the position detection system 4 includes a game casing 70 having a display unit 73 and a speaker 74, a gun-type imaging device 71 that imitates a gun, a shield-type imaging device 72 that imitates a shield, Is provided.
[0171]
An animation image or the like that is the content of the game is output to the display unit 73, and sound effects, music, and the like are reproduced from the speaker 74 in accordance with the image displayed on the display unit 73. In the position detection system 4 shown in FIG. 26, a background image, an identifier 75, an aiming point 76, and an aiming point 77 are displayed on the display unit 73, although details are not shown.
[0172]
In this way, the position detection system 4 determines the aim point 76 toward the shooting target displayed on the display unit 73 using the gun-type imaging device 71 as an operation input device imitating the gun 1000, for example. The game device is configured such that the aiming point 77 as a shield is set using the shield-type imaging device 72 as an operation input device imitating a shield to prevent flying objects.
[0173]
The gun-type imaging device 71 is an imaging camera including a CCD (Charge Coupled Device) as an imaging device, and displays an image displayed on the display unit 73 when the user 1000 faces the display unit 73 of the game housing 70. The surrounding environment including the image is captured, and the image in the imaging range 78 is transmitted to the game housing 70. At this time, the center of the imaging range 78 captured by the gun-type imaging device 71 is displayed as a sighting point 76 at a corresponding location on the display unit 73.
[0174]
The shield-type image pickup device 72 is also an image pickup camera having a CCD, and picks up the surrounding environment including an image displayed on the display unit 73 when facing the display unit 73 by the user. An image in the range 79 is transmitted to the game housing 70.
At this time, a predetermined area at the center of the imaging range 79 is displayed as a sighting point 77 at a corresponding portion of the display unit 73.
[0175]
Since the position detection system 4 performs the same processing as the position detection system 1 of the first embodiment, the specific circuit configuration is the same as those of the position detection system 1 of the first embodiment. Detailed description will be omitted.
[0176]
As a matter of course, even components that are common to the position detection system 1 of the first embodiment can be appropriately modified so as to be adapted to the position detection system 4 of the fourth embodiment. Needless to say, it can be done.
[0177]
Since the process of displaying the aiming point is the same as that of the position detection systems 1 to 3 described above, a description thereof will be omitted.
[0178]
The position detection system 5 shown as the fifth embodiment of the present invention displays an identifier on a display device, captures a display image including the identifier with an imaging device as an operation input device, and analyzes the captured image. A position detection system that detects the positional relationship between the display device and the imaging device, the orientation of the imaging device with respect to the display device, and the like by non-contact by analyzing in the means, Alternatively, it is a user interface that allows an imaging device as an operation input device to face a predetermined image and can perform operation input by a position detection operation using an asynchronous shutter.
[0179]
As shown in FIG. 27, the position detection system 5 includes a game housing 80 having a display unit 82 and a speaker 83, a racket-type imaging device 81 that imitates a racket, and a contact sensor 84.
[0180]
An animation image or the like that is the content of the game is output to the display unit 82, and sound effects, music, and the like are reproduced from the speaker 83 in accordance with the image displayed on the display unit 82. In the position detection system 5 shown in FIG. 27, a background image, an identifier 85, and a sphere 86 are displayed on the display unit 82, although details are not shown.
[0181]
In this way, the position detection system 5 is displayed on the display unit 82 by performing an operation of actually hitting a ball using the racket type imaging device 81 as an operation input device imitating the racket, for example. The game device is configured such that the ball 86 is returned to the player.
[0182]
Since the position detection system 5 performs the same processing as the position detection system 1 of the first embodiment, the specific circuit configuration is the same as that of the position detection system 1 of the first embodiment. Detailed description will be omitted.
[0183]
The user holds the racket-type imaging device 81 imitating a table tennis racket in accordance with a sphere displayed with a three-dimensional depth on the display unit 82, and performs an operation of hitting the displayed sphere. At this time, an image in the imaging range 87 is captured and transferred to the game housing 80.
[0184]
The game control unit provided in the game housing 80 detects the position and orientation of the racket-type imaging device 81 at a timing corresponding to the time when the sphere 86 displayed with a three-dimensional depth on the display device reaches the user. Is detected from the actually detected position and angle, and the trajectory of the hit ball is calculated and displayed so that the ball 86 is hit according to the trajectory.
[0185]
Further, for example, the user strongly steps on the contact sensor 84 during smashing. At this time, the position detection by the asynchronous shutter is performed using the timing when the contact sensor 84 is stepped on as a trigger.
[0186]
Therefore, such a position detection system 5 can accurately detect the orientation and angle of the racket-type imaging device 81 while performing high-speed movement. The position detection system 5 can be applied not only to table tennis smash but also to similar movements such as tennis smash, golf shot, baseball batting, and dance step.
[0187]
As a matter of course, even components that are common to the position detection system 1 of the first embodiment can be appropriately modified so as to be adapted to the position detection system 5 of the fifth embodiment. Needless to say, it can be done.
[0188]
In the conventional method of detecting the scanning line of the cathode ray tube with the light receiving element, it is necessary to flash the screen every time the position is detected. Therefore, it is necessary to always detect the position and continuously display the pointer. Although not suitable, position detection at a sampling rate higher than the conventional position detection system as shown in the position detection system 4 and the position detection system 5 can be realized by making full use of the above-described multiple exposure photographing.
[0189]
Accordingly, in such a motion detection operation, the determination error range of motion detection can be within the range of human (user) sensory error, so that the aiming point 77 is always displayed as if it were a pointer for the user. It is possible to be as it is. In addition, shooting with an asynchronous shutter enables accurate position detection even for high-speed movements, thereby realizing comfortable operation input that does not give the user a sense of incongruity.
[0190]
As described above, according to the position detection system shown as the first to fifth embodiments, the movement similar to the movement of the game player (user) or the movement in the game world is used as the input information. For example, it is possible to reflect a real-world motion in a virtual world such as a game, and motion detection with higher accuracy and improved real-time control becomes possible.
[0191]
The position detection apparatus as described above can also be applied to projectors, liquid crystal televisions, VGA monitors, and the like that have been introduced in general homes, and is not affected by the specifications of scanning lines such as NTSC and PAL.
[0192]
In the position detection systems 1 to 5, the identifier may not be normally displayed. That is, only when there is an input from the imaging device as the operation input device, for example, in the case of a shooting game, the identifier can be displayed only when the trigger of the gun-type controller is pulled. is there.
[0193]
In addition, each exposure timing at the time of multiple exposure shooting or each shutter timing at the time of shooting by an asynchronous shutter can be used as the timing for displaying the identifier.
[0194]
In the position detection systems 1 to 5, the game processing unit can perform the extraction of feature points by performing chroma key processing. The chroma key process may be performed by the image processing unit.
[0195]
It should be noted that the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present invention.
[0196]
【The invention's effect】
As described above in detail, the position detection device and the position detection method according to the present invention displays a plurality of identifiers, images the identifiers via the imaging unit, and includes the captured identifiers. Is used to analyze the position of the imaging means relative to the display means.
[0197]
Therefore, according to the position detection device and the position detection method as described above, the real world operation can be performed in the virtual world by detecting the actual operation similar to the operation in the virtual world such as the displayed image without contact. It is possible to reflect the motion, and it is possible to perform motion detection with higher accuracy and improved real-time control regardless of the type of display device.
[0198]
Further, the position detection device and the position detection method as described above can be applied regardless of the type of display device. Therefore, it can be applied to projectors, liquid crystal televisions, VGA monitors, and the like that have begun to be introduced in ordinary homes, and is not affected by the specifications of scanning lines such as NTSC and PAL. Therefore, the cost at the time of introduction is reduced, and convenience for the user is improved.
[0199]
The entertainment apparatus according to the present invention includes a display control unit that performs control to display a plurality of identifiers, an imaging unit that captures the identifiers via an imaging unit, and the plurality of imaging units that use the captured identifiers. Analysis means for analyzing the position relative to the display means for displaying the identifier, and control means for controlling the program by converting the position information analyzed by the analysis means into control information.
[0200]
Therefore, according to the entertainment apparatus as described above, the real world can be detected in the virtual world by detecting the actual movement similar to the movement of the game player (user) and the virtual world such as the game world in a non-contact manner. It is possible to reflect the motion, and it is possible to perform motion detection with higher accuracy and improved real-time control regardless of the type of display device. Furthermore, it is possible to control the program by converting the detected position information into control information.
[0201]
In addition, the entertainment device as described above can be applied regardless of the type of display device. Therefore, it can be applied to projectors, liquid crystal televisions, VGA monitors, and the like that have begun to be introduced in ordinary homes, and is not affected by the specifications of scanning lines such as NTSC and PAL. Therefore, the cost at the time of introduction is reduced, and convenience for the user is improved.
[Brief description of the drawings]
FIG. 1 is a schematic diagram showing an outline of a configuration of a position detection system shown as a first embodiment of the present invention.
FIG. 2 is a block diagram showing a configuration of a position detection system shown as the first embodiment of the present invention.
FIG. 3 is a schematic diagram showing an outline when the position detection system shown as the first embodiment of the present invention performs calibration.
FIG. 4 is a schematic diagram showing an outline when the position detection system shown as the first embodiment of the present invention performs calibration.
FIG. 5 is a flowchart showing a process in which an image is captured by the control type imaging apparatus in the position detection system shown as the first embodiment of the present invention.
FIG. 6 is a flowchart showing processing in which the game control unit in the position detection system shown as the first embodiment of the present invention detects the position of the aiming point.
FIG. 7 is a schematic diagram showing a concept for predicting the position of an identifier.
FIG. 8 is a schematic diagram showing an outline of a configuration of a position detection system shown as a second embodiment of the present invention.
FIG. 9 is a block diagram showing a configuration of a position detection system shown as a second embodiment of the present invention.
FIG. 10 is a flowchart illustrating a process in which an image is captured by a control-type imaging device in the position detection system according to the second embodiment of the present invention.
FIG. 11 is a flowchart showing processing in which the game control unit detects the position of the aiming point in the position detection system shown as the second embodiment of the present invention.
FIG. 12 is a flowchart showing processing performed by an image control unit in the position detection system shown as the second embodiment of the present invention.
FIG. 13 is a block diagram showing a configuration of a position detection system shown as a third embodiment of the present invention.
FIG. 14 is a conceptual diagram showing the concept of a vector field.
FIG. 15 is a characteristic diagram used to explain a procedure for calculating a motion vector from a density change.
FIG. 16 is a flowchart showing processing in which the game control unit in the position detection system shown as the third embodiment of the present invention detects the position of the aiming point.
FIG. 17 is a block diagram illustrating a configuration of an imaging apparatus that performs imaging using an asynchronous shutter and multiple exposure imaging.
FIG. 18 is an explanatory diagram illustrating a light receiving / charge transfer unit in a solid-state imaging unit of an imaging apparatus that performs imaging using an asynchronous shutter and multiple exposure imaging.
FIG. 19 is a diagram illustrating a part of a light receiving / charge transfer unit in a solid-state imaging unit of an imaging apparatus that performs imaging using an asynchronous shutter and multiple exposure imaging.
FIG. 20 is a time chart for explaining the operation of the imaging apparatus that performs imaging using an asynchronous shutter and multiple exposure imaging.
FIG. 21 is an enlarged view of the time chart showing a part of the time chart shown in FIG. 20 in an enlarged manner.
FIG. 22 is a time chart for explaining the operation of the imaging apparatus that performs imaging using an asynchronous shutter and multiple exposure imaging.
FIG. 23 is a time chart for explaining the operation of the imaging apparatus that performs shooting using an asynchronous shutter and multiple exposure shooting.
FIG. 24 is a schematic diagram showing an image that has been subjected to multiple exposure shooting of N = 2.
FIG. 25 is a diagram illustrating imaging timing when imaging is performed at an arbitrary timing by an asynchronous shutter.
FIG. 26 is a schematic diagram showing an outline of a configuration of a position detection system shown as a fourth embodiment of the present invention.
FIG. 27 is a schematic diagram showing an outline of a configuration of a position detection system shown as a fifth embodiment of the present invention.
[Explanation of symbols]
1, 2, 3, 4, 5 Position detection system 10, 70, 80 Game housing, 11, 60 Controller-type imaging device, 12, 73, 82 Display unit, 13, 74, 83 Speaker, 14, 75, 85 Identifier, 15 target, 16 aiming point, 17, 78, 79 imaging range, 20 lens unit, 21 CCD imager, 22 S / H unit, 23 AGC unit, 24 A / D unit, 25 signal processing unit, 26 TG unit, 27 imaging drive unit, 28 imaging control unit, 29 SDRAM, 30 main memory, 31 mechanical I / F, 32 game control unit, 33 display control unit, 34 ROM, 35 image display unit, 36 frame memory, 37 bus, 40 Monitor, 50 game device, 51 I / F unit, 52 Monitor I / F unit, 53 Vector operation unit, 61 I / F unit, 71 Gun type imaging device, 72 Shield type imaging device, 76 , 77 Aiming point, 81 Racket type imaging device, 84 Contact sensor, 86 spheres

Claims (15)

Analyzing means for analyzing the position of the imaging means relative to the display means using an identifier imaged by an imaging means for imaging the identifier displayed on the display means ;
Storage means for storing the position of a predetermined point within the imaging range imaged in the past ,
The analysis means uses the image taken by multiple exposure by the imaging means, and the temporal relationship between a plurality of identifiers displayed in one image is stored in the storage means in the past. A position analysis device that analyzes the position by analogy with the fact that the position change is continuous .   The position analysis apparatus according to claim 1, wherein the imaging unit includes an asynchronous shutter.   The position analysis apparatus according to claim 1, wherein the analysis unit is wirelessly connected to the imaging unit. It further comprises vector calculation means for performing motion detection,
The position analysis apparatus according to claim 1, wherein the analysis unit performs motion detection by the vector calculation unit.   The position analysis apparatus according to claim 1, further comprising display means for displaying the identifier.   The position analysis apparatus according to claim 1, further comprising an imaging unit that images the display unit.   The position analysis device according to claim 1, wherein the identifier is an identifier identified by a predetermined color range.   The position analysis device according to claim 1, wherein the identifier is an identifier identified by a predetermined pattern.   The position analysis device according to claim 1, wherein the identifier is an identifier identified by a predetermined luminance range.   The position analysis apparatus according to claim 1, wherein the analysis unit automatically recognizes the identifier.   11. The position analyzing apparatus according to claim 10, wherein the analyzing means recognizes the imaged identifier by comparing with a position of the identifier detected by a predetermined operation determined in advance as a reference.   12. The position analyzing apparatus according to claim 11, wherein the predetermined operation is an operation controlled by the analyzing means.   The position analyzing apparatus according to claim 11, wherein the predetermined operation is an operation synchronized with the analyzing means. An analysis step of analyzing the position of the imaging means relative to the display means using an identifier imaged by an imaging means for imaging the identifier displayed on the display means ;
A storage step of storing the position of a predetermined point in the imaging range captured in the past in the storage means ,
In the above analysis process, using an image that is multiple exposure shooting by the image pickup means, a temporal context between a plurality of identifiers to be displayed in one image, past the predetermined point stored in the storage means A position analysis method for analyzing the position by analogy with the fact that the position change is continuous . Analyzing means for analyzing the position of the imaging means relative to the display means using an identifier imaged by an imaging means for imaging the identifier displayed on the display means;
Storage means for storing the position of a predetermined point within the imaging range imaged in the past;
Control means for controlling the program by converting the position information analyzed by the analysis means into control information,
The analysis means uses the image that has been subjected to multiple exposure photography by the imaging means, and the temporal relationship between a plurality of identifiers displayed in one image is stored in the storage means in the past. An entertainment device that analyzes the position by analogy with the fact that the position change is continuous .

Images