JPH11316645A - Information input device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a practical three-dimensional(3D) information input device capable of easily performing change of pointing or the view point in a 3D space and naturally moving the character or the like of animation while using the gesture or action of a user as it is. SOLUTION: Concerning the device for providing the differential image of an object at the time of irradiation and no irradiation with light, this device is provided with a light emitting means 3 for irradiating an object with light, area image sensor (IMS) 1 having image pickup parts constituted by two- dimensionally arranging plural light receiving elements for photoelectric conversion and a CCD type charge transfer means or transferring and extracting charges provided by these image pickup parts, and timing signal generating means 5 for driving the CCD type charge transfer means of the IMS, controlling the timing of charge transfer from the light receiving element to the CCD type charge transfer means and performing control the amount of light receiving charges at the time of light emission at the light emitting means and the amount of light receiving charges at the time of light non-emission can be arranged as predetermined in all the IMS or respective CCD type charge transfer means.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an information input device for performing pointing in a three-dimensional space using an image.

[0002]

2. Description of the Related Art As an input device for a computer, in particular, a pointing input device, a mouse which is almost standard equipment is overwhelmingly used. However, what the mouse can do is move the cursor, select a menu, and so on, and serve only as a two-dimensional pointing device.

Since it is two-dimensional information that can be handled by a mouse, it is difficult to select an object having a depth such as an object in a three-dimensional space. Also, when creating an animation, it is difficult to make a natural movement with a mouse even when it is desired to use the character to make a movement.

[0004] To compensate for such difficulties in pointing in a three-dimensional space, various three-dimensional pointing devices have been developed. <Three-dimensional pointing device [Part 1]> As a typical three-dimensional pointing device, for example, FIG.
There are the following. In this three-dimensional pointing device, the center round operation knob means section 150 is "pushed forward", "presses the center", "presses backward", "lifts" the entire round operation knob means section 150,
There are six types of operations, such as "turn the whole clockwise" and "turn left", with six degrees of freedom.

By assigning the six degrees of freedom to various operation instructions, the position (x, y, z) and direction (x axis, y axis, z axis) of the cursor in the three-dimensional space can be controlled, or Viewpoint position (x, y, z) and direction (x axis,
(y-axis, z-axis) can be controlled.

However, when this is actually operated, there is a problem that the cursor and the viewpoint cannot be controlled as desired. For example, when trying to turn to the left or right, the user pushes forward or backward, and the cursor moves in an unexpected direction or the viewpoint moves.

[0007] Devices capable of inputting to such a three-dimensional pointing device by using gestures or gestures have been developed. This is introduced next.

<Three-dimensional pointing device [No. 2]> It is, for example, a data glove, a data suit, or a cyber glove. These are, for example, data gloves that are glove-like devices with optical fibers running over the surface. The optical fiber passes through the joint of the finger, and when the finger is bent, the conduction of light changes. By measuring the light conduction, it is possible to determine how much the finger joint is bent. The position of the hand itself in the three-dimensional space is measured by a magnetic sensor on the back of the hand. If you decide your gesture and the corresponding instructions, such as moving your index finger forward, you can use a data glove to change your viewpoint in 3D space and just walk around (walkthrough) Can be called).

However, there are some problems. First, the price is expensive and it is difficult to use it for home use.

Second, there is an erroneous recognition of the operation. That is, since the angle of the finger joint is measured, for example, only the index finger is extended, and other fingers may be erroneously recognized as other instructions even if the bent state is defined as a forward instruction. In other words, even if you say that you extend your finger to a bite, it is rare that the angle of the second joint of the index finger is perfect at 180 degrees. Is difficult to recognize as stretched.

Thirdly, since the user wears the data glove, natural operation is hindered. 4th
In this method, the light conduction state must be calibrated between the open state and the closed state of the hand each time it is worn, so that it cannot be used easily.

Fifth, there is a problem of failure. Because it uses optical fiber, if you use it continuously,
The problem is that the device is close to a consumable, such as a broken fiber.

Sixth, even though the device is expensive and time-consuming, if the size of the gloves is not exactly the same, the gloves may be shifted during use and calibrated. The point is that it is difficult to recognize a small hand gesture because of the deviation.

As described above, due to various problems, the data glove is a virtual reality (VR).
Despite being a trigger of the (virtual reality) technology, it is not as widespread as expected at the beginning and the price has not been reduced, and there are many problems in terms of usability.

On the other hand, some attempts have been made to input a hand gesture or gesture without wearing a special device such as a data glove. An example is given below.

<Three-dimensional pointing device [3]> A typical example of an attempt to input a hand gesture or a gesture is a method of analyzing a moving image such as a video image and recognizing a hand shape. .

However, in this method, in the case of recognizing a target image and a hand gesture from a background image, it is necessary to cut out only a hand, but it is very difficult to cut out the target image portion. There is a problem.

For example, consider a case in which a "hand" as a target image is cut out using colors. Since the color of the hand is flesh color, a method of cutting out only the flesh color image area can be considered. However, this is unrealistic because it is difficult to identify the skin color when there are beige clothes or walls in the background. Also, even if you make adjustments so that you can distinguish beige from skin color, if the lighting changes,
Since the color tone changes, it is difficult to cut out constantly.

In order to avoid such a problem, a measure has been taken to limit the background image, such as placing a blue mat on the background, to facilitate clipping. Alternatively, a measure has been taken such that a fingertip is colored so as to be easily cut out from the background, or a colored ring is worn. However, such limitations are not practical and have been used experimentally, but have not yet been put to practical use.

The video image recognition processing such as the above-described clipping requires a large amount of calculation. For this reason, the current state of the art is that a current personal computer cannot process a video image generated as many as 30 frames per second. Therefore, it is difficult to perform a motion capture or the like by processing a video image in real time.

<Three-Dimensional Pointing Device [Part 4]> As another method for analyzing a moving image such as a video image and inputting a hand gesture or a gesture, there is a method using a range finder device for inputting a distance image. .

A typical principle of the range finder is to irradiate a spot light or a slit light to a target object and to obtain a triangulation principle from a light receiving position of the reflected light. In the range finder, spot light or slit light is mechanically scanned in order to obtain two-dimensional distance information. Although this device can generate a highly accurate range image, the configuration of the device is large and the cost is high. Also, it takes time to input, and it is difficult to perform processing in real time.

<Three-dimensional pointing device [part 5]> As another method of inputting a hand gesture or a gesture by analyzing a moving image such as a video image, a color marker or a light emitting part is attached to a part of a hand or a body, and an image is displayed. There is also the use of a device that detects them by means of the device and captures the shape and movement of the hand / body. This device has been partially put into practical use. However, considering the convenience of the user, it is a great disadvantage that the device must be mounted every time the operation is performed, and the range of application is greatly restricted. Also, as seen in the data glove example,
A device that is used by attaching the device to a movable part such as a hand tends to have a problem of durability.

As described above, there are various types of three-dimensional pointing devices, but a system that can be used in the future does not require the operator to wear the device or the operator to operate the device directly. This is considered to be a method of analyzing and using a moving image such as a video image. Therefore, the conventional camera technology will be considered.

<Consideration of conventional camera technology> In the conventional camera technology, in order to combine a character with a background (chroma key), a character is previously photographed with a blue background, and the character is easily cut out. I needed to. For this reason, there were restrictions on the shooting location, such as in a studio where blue-back shooting is possible. Alternatively, in order to cut out a character from a video shot in a non-blue-back state, the character cutout range has to be manually edited for each frame, which is extremely time-consuming.

Similarly, in order to generate a character in a three-dimensional space, a method is used in which a three-dimensional model is created in advance, and a picture of the character is pasted (texture mapping) there. . However, the generation of the three-dimensional model and the texture mapping are troublesome and have little utility except for applications that may be expensive, such as movie production.

In order to solve such a problem, for example, there is a technique as disclosed in Japanese Patent Application Laid-Open No. H10-17749. This is a method of extracting a reflected light image and acquiring a distance image. However, this method has a problem that a commercially available sensor array cannot be used.

[0028]

As described above, there are various types of three-dimensional pointing devices, but a system that can be used in the future does not require the operator to wear the device or operates the device. This method is considered to be a method of analyzing and using a moving image such as a video image, which does not need to be directly operated by a user.

However, in the conventional camera technology, in order to combine a character (chroma key) with the background, it was necessary to photograph the character in advance with a blue background and to easily cut out the character. There were restrictions on available shooting locations, such as studios where you can shoot in the blue background.

Further, in order to cut out a character from a video shot in a state other than the blue background, the character cutout range must be manually edited for each frame, which is extremely time-consuming and impractical.

Similarly, in order to generate a character in a three-dimensional space, a method is used in which a three-dimensional model is created in advance, and a picture of the character is pasted (texture mapping) there. . But,
The generation of the three-dimensional model and the texture mapping are troublesome, and have little utility except for applications that may be expensive, such as movie production.

There is also a method of extracting a reflected light image and acquiring a distance image, but this method has a problem that a commercially available sensor array cannot be used. As described above, in recent years, the necessity and demand for performing three-dimensional input have increased, but there has been no direct instruction type input device capable of easily inputting a gesture or movement without mounting a special device. .

Therefore, development of a practical and simple three-dimensional input device capable of easily changing a pointing and a viewpoint in a three-dimensional space has been demanded. Therefore, an object of the present invention is to make it possible to easily perform pointing and change of a viewpoint in a three-dimensional space, and to give a natural movement to an animation character or the like using a user's gesture or movement as it is. It is an object of the present invention to provide a practical and practical three-dimensional information input device.

[0034]

In order to achieve the above object, the present invention is configured as follows. That is, in an apparatus for obtaining a difference image between a subject image at the time of light irradiation and a subject image at the time of no light irradiation, a light emitting unit for irradiating the subject with light, and an imaging unit having a plurality of two-dimensionally arrayed light receiving elements for photoelectric conversion An area image sensor having CCD-type charge transfer means for transferring and taking out the charge obtained by these image pickup units; driving of the CCD-type charge transfer means of the area image sensor and charge from the light receiving element to the CCD-type charge transfer means; The charge amount received when the light emitting unit emits light and the charge amount received when the light emitting unit does not emit light are arranged in a predetermined array in all or each CCD type charge transfer unit. And timing signal generating means for controlling the signals in a line.

According to this configuration, the pixels of the subject image at the time of light emission and at the time of non-light emission are alternately arranged in pixel units by timing control from the two-dimensionally arranged light receiving elements of the CCD area image sensor. Images are directly transferred to the CC
It can be obtained from a D-type area image sensor, and a difference image can be obtained in real time by extracting a difference between pixels, so that a reflection image of a hand or the like can be obtained easily and in real time. . This makes it unnecessary to cut out the image of the object, which is the most difficult in the conventional image processing and has been a barrier to the application of the image processing. Therefore, according to the present invention, it is possible to easily and stably provide various image processing which has been difficult to put into practical use at a low cost by using commercially available components. It will bring great changes to a wide range of markets.

[0036]

Embodiments of the present invention will be described below with reference to the drawings. (First Embodiment) FIG. 1 is a block diagram showing a schematic system configuration of a first embodiment of the present invention.

In FIG. 1, reference numeral 1 denotes an image pickup device, which uses a CCD area image sensor as shown in FIG. 2 is light emission control means, 3 is LE as a light source
D (light emitting element), 4 is an optical system, 5 is a timing generator, 6 is a difference circuit, 7 is an analog amplifier, 8 is an analog / digital converter, 9 is a system controller, 10 is a frame buffer, and 11 is a personal computer. An interface circuit.

In this system, a CCD area image sensor 21 as shown in FIG. 2 is used as the image pickup device 1 for acquiring an image. The CCD type area image sensor 21 includes a plurality of light receiving elements 22 (22-a1,..., 22-ni) arranged in order in a row direction and a column direction, and these light receiving elements 22 (22-a1,. ni), the vertical charge transfer means 23 (23-a, to 23) for transferring charges from the light receiving element groups arranged in the vertical direction.
-N) and the vertical charge transfer means 23 (23-a,
, 23-n), the horizontal direction charge transfer means (image signal output charge transfer means) 2 for reading out the transferred charges in line units and transferring them in the horizontal direction so as to output them as image signals.
4, and transfers the charges obtained by the respective light receiving elements constituting the pixel to the vertical charge transfer means 23 (23-a to 23-a to 23-a).
n), and this is transferred by the horizontal charge transfer means 24, which is a charge transfer means for outputting an image signal, to obtain an image signal in line units.

In particular, the CCD area image sensor 21 used in the present system in the first embodiment has the number of stages of the vertical charge transfer means 23, that is, twice the number of pixels that the vertical charge transfer means 23 can hold. An element having a horizontal direction charge transfer means 24 having a number of stages (double the number of pixels) is used. Then, taking advantage of the fact that charge transfer can be controlled for each odd-numbered line, the odd-numbered line is, for example,
The even lines are used, for example, to acquire an image of a subject not illuminated with illumination light, and the charges in the illuminated image without illumination are alternately arranged in the vertical direction charge transfer means 23. At the same time, the horizontal charge transfer means 24 having a capacity to hold twice the number of lines of the vertical charge transfer means 23 is provided to the two vertical charge transfer means 23 by two pixels ( (Two charges) can be transferred and held in the horizontal direction charge transfer means 24.

For this reason, in the present system, the gate for transferring the charge from each light receiving element to the corresponding vertical charge transfer means 23 can be individually controlled for each light receiving element. A CCD type area image sensor generally called a progressive CCD type area sensor having the above-mentioned format is used.

The timing generator 5 is for generating a timing control signal such as a pulse signal or a modulation signal. The light emitting means 3 is a light source for illuminating a subject, and is composed of, for example, an LED. . The light emission control means 2 controls light emission of the light emission means 3 and controls light emission of the light emission means 3 in synchronization with a timing signal generated by the timing generator 5.

The optical system 4 collects the reflected light that is reflected by the light from the light emitting means 3 and hits the object and bounces off the object.
Is a lens for forming an image on the light receiving surface.

The difference circuit 6 is a circuit for obtaining a difference between an image obtained when the light emitting means 3 emits light and an image obtained when the light emitting means 3 does not emit light.
This is a circuit for calculating a difference in electric charge between the light receiving elements of each pixel of the D-type area image sensor 1.

The analog amplifier 7 is connected to the difference circuit 6
A circuit for amplifying the charge signal obtained in
The D converter 8 is a circuit for converting the charge signal amplified by the analog amplifier 7 into digital data. The frame buffer 10 temporarily stores the digitalized image signal in frame units. Memory.

The interface circuit 11 is for transmitting the signal stored in the frame buffer 10 to the main unit of the processing device such as a PC (personal computer). The system control means 9 includes the light receiving control means 2 and the timing generator. 5, for controlling various controls of the A / D converter 8, the frame buffer 10, the interface circuit 11, and the like.

Since the detailed driving method of the CCD area image sensor 1 is described in detail in "Toshiba CCD Area Image Sensor Data Book" and the like, the operation of the present invention will be described here in order to make it easy to understand. Only the parts necessary to explain are described.

<Configuration Example of CCD Area Image Sensor 1> FIG. 2 is a plan view showing a schematic configuration example of the area image sensor 21. As shown in FIG. 2, the CCD area image sensor 21 includes a plurality of light receiving elements 22 (22-a1, 22-a1,
22-a2, 22-a3, ~, 22-ai, ~, 22-
n1, 22-n2, 22-a3,..., 22-ni), the vertical charge transfer means 23 (23-a, 23-b,.
23-n), and a horizontal direction charge transfer means 24.

Of these, the light receiving element 22 (22-a
1,22-a2,22-a3, ~, 22-ai, ~, 2
2-n1, 22-n2, 22-a3, ~, 22-ni)
Is an arrangement of photodiodes (Photodiodes: PD) arranged two-dimensionally, that is, in a matrix manner. Here, a1, a2, a
Subscripts 3, 3,, ai,, n1, n2, a3,, and ni are attached to indicate the position of the light receiving element. The charge transfer means is a charge transfer means 2 for the vertical direction.
3 (23-a, 23-b, 23-c,..., 23-n) and charge transfer means 24 for the horizontal direction.
This part is an analog shift register constituted by a CCD (Charge Couple Device).

The vertical charge transfer means 23 (23-
a, 23-b, 23-c,..., 23-n) are arranged one by one for each light receiving element group unit arranged in the vertical direction (Y-axis direction). In this example, the charge transfer means 23 for the vertical direction is used.
-A is the light receiving element 22-a1, 22-a2, 22-a3
, And 22-ai, and the vertical charge transfer means 23-b includes light receiving elements 22-b1, 22-b2,
22-b3, ..., 22-bi charge transfer,
.., The vertical charge transfer means 23-n is a light receiving element 22.
-N1, 22-n2, 22-n3,..., 22-ni.

The horizontal charge transfer means 24 comprises vertical charge transfer means 23-a, 23-b and 23-.
c,..., 23-n, the electric charge transferred from one stage (one line, that is, the accumulated electric charge from n light receiving elements when viewed in a line in the X-axis direction) ) Is received and output to the outside as an output of the CCD image sensor 21.

The CCD image sensor 2 having the above configuration
Various control signals are required to drive the control signal 1, and these control signals are generated by the above-described timing generator 5.

The CC generated by the timing generator 5
As shown in FIG. 2, signals given to the D image sensor 21 are φ1, φ2, φ3, φ4, φ5, φ6, G1, G2.
There are eight types.

Of these, φ1, φ2, and φ3 perform vertical charge transfer, φ4, φ5, and φ6 perform horizontal charge transfer, and G1 and G2 perform charge transfer from the light receiving element to the charge transfer means. It controls the opening and closing of the gate to be controlled. G1 controls odd lines, and G2 controls even lines.

<Description of Operation> Next, the operation of the present system in the first embodiment having such a configuration will be described with reference to FIGS. In this system, in order to extract the image of the target portion of the subject, an image of the subject in the case of no illumination and an image of the subject in the case of illumination are obtained, and both are obtained without using a frame memory or the like. The reflected light of the target part is extracted, and the operation will be described. FIG. 8 is a flowchart of the process.

Prior to the processing, the two-dimensional array of light receiving elements 22
The light receiving portion of the CCD image sensor 21 composed of
2-a1, 22-b1, 22-c1,..., 22-n1, a first line including a total of n light receiving elements, 22-a2, 2
2-b2, 22-c2,..., 22-n2, a second line composed of a total of n light receiving elements, 22-a3, 22-b3, 2
A third line composed of a total of n light receiving elements 2-c3,..., 22-n3,..., 22-ai, 22-bi, 22-c
i,..., an i-th line composed of a total of n light receiving elements of 22-ni, i.
The main lines are formed, and the charges obtained by the light receiving elements from the first line to the n-th line are transferred to the vertical charge transfer means 23-a, 23-b,
23-c,..., And 23-n.
The charge of one line is transferred in the order of line, .about., Third line, second line, and first line. At this time, the horizontal direction charge transfer means 24 is operated to shift, so that the image is transferred line by line. A signal can be output.

Assuming that i is an odd integer, the i-th line, the (i-2) th line,..., The third line, and the first line are the odd-numbered lines of the television scan, and the (i-1) th line and the i-th line. The third line, the fourth line, and the second line correspond to the even-numbered lines of the television scanning.

Then, in order to be able to discriminate whether the image signal output from the horizontal charge transfer means 24 is a signal obtained by reading an even-numbered line or an odd-numbered line, The CCD area image sensor 21 has a flag C output. Each of the light receiving elements 22 and the vertical direction charge transfer means 23-a, 23-b,
There are clear terminals for clearing the contents (holding charges) of the charge transfer means 24 for the horizontal direction, and 23-c,..., 23-n.

Therefore, first, the flag C is cleared and each light receiving element 22 and the vertical charge transfer means 23-a, 23
-B, 23-c,..., 23-n and the horizontal direction charge transfer means 24 are subjected to an initialization process of giving a clear signal (step S100).

Next, in order to illuminate and image the subject, the light emission control means 2 causes the light emitting element (LED) 3 for subject illumination to emit light (step S101). The light emitted by this is reflected by an object, which is a subject, and a lens (light emitting system) 4
And is incident on the CCD area image sensor 21 constituting the imaging unit 1 to form an optical image on the light receiving elements 22 in the two-dimensional array. The formed optical images are light receiving elements 22-a1, 22-a2,.
22-n1,..., And 22-ni, are converted into charges proportional to the intensity of the reflected light and accumulated. Thus, each light receiving element 22-a1, 22-a2,..., 22-n1,..., 22 of the CCD area image sensor 21 constituting the imaging unit 1
-Ni means that a charge per pixel at that pixel position is obtained.

By applying a voltage to G1, the light receiving element 22
-A1, 22-a2, ~, 22-n1, ~, 22-ni
Of the light receiving elements 22-a1, 22-a of the odd-numbered lines
3, ~, 22-an, ~, 22-n1,22-n3
, 22-ni to vertical charge transfer means 23-a,
Activate the gates to 23-b, 23-c, ..., 23-n.

Then, the light receiving element 22-a of the odd-numbered line
1,22-a3, ~, 22-an, ~, 22-n1,2
Only the charges accumulated in 2-n3,..., 22-ni are transferred to the vertical charge transfer means 23-a, 23 corresponding to each light receiving element.
-B, 23-c, ~, 23-n.
(Step S102). At this time, the light receiving elements 22-a2, 22-a4,.
n-1, ~, 22-n2, 22-n4, ~, 22-ni
-1 is transferred to the vertical charge transfer means 23-.
a, 23-b, 23-c, ..., 23-n are not transferred at all. Therefore, the state is as shown in FIG. Here, e indicates the charge for each pixel.

Thereafter, the CCD area image sensor 21
Light receiving elements 22-a1, 22-a2, ..., 22-n
1 to 22-ni are reset once and each light receiving element 2
2-a1, 22-a2, ~, 22-n1, ~, 22-n
Clear the charge stored in i (step S10)
3).

Thus, the light receiving element 22-a of the odd line
1,22-a3, ~, 22-an, ~, 22-n1,2
As for the electric charges accumulated in 2-n3,.
, 23-n. At this time, the position of the even-numbered line is empty in the vertical direction charge transfer means 23-a, 23-n.

Next, the light emission control means 2 stops the light emission of the light emitting element (LED) 3 so as to image the object without illumination (step S104). Then, the light receiving elements 22-a2, 22-a4,.
1, ~, 22-n2, 22-n4, ~, 22-ni-1
Only the electric charge accumulated in the vertical charge transfer means 23
Apply a voltage to G2 for transfer to -a, 23-b, 23-c, ..., 23-n. Thereby, the vertical direction charge transfer means 23-a, 23-b, 23-c,.
The gate to 23-n becomes active.

Then, as shown in FIG. 4, the accumulated electric charge during non-light emission is changed to the light receiving elements 22-a2, 22-a4,
~, 22-an-1, ~, 22-n2,22-n4
, 22-ni-1 to the corresponding vertical charge transfer units 23-a, 23-b, 23-c,..., 23-n (step S105).

As a result, the charges from the light receiving elements of the even lines are transferred to the positions corresponding to the even lines of the vacant vertical charge transfer means 23-a,..., 23-n, as can be seen from FIG. In this state, the vertical charge transfer means 2
In 3-a, 23-b, 23-c,..., And 23-n, charges proportional to the intensity of reflected light during light emission and non-light emission are alternately accumulated.

Next, the vertical charge transfer means 23-a,
The charges accumulated in 23-b, 23-c,..., 23-n are transferred one step (shifted by one pixel) (step S1).
07). This transfer operation can be performed by applying a voltage to φ1, φ2, and φ3 as shown in FIG. In our example,
The charge at the time of non-emission stored in the lowermost stage in FIG. 4 is transferred to the horizontal charge transfer means 24, and is accumulated as shown in FIG.

Next, it is checked whether or not the even line has already been read (step S108). In other words, if the flag C is "0", it is not set that the even-numbered line has been read out yet.
The charge accumulated in the horizontal charge transfer means 24 is
A voltage is applied to φ4, φ5, and φ6 to transfer (step S109).

At this time, as shown in FIG. 6, the electric charge at the time of non-emission is accumulated at a place shifted one step to the left. Next, since the current even line has been read, the fact is set in the flag C (step S110).

After the flag is set, the charges of the vertical charge transfer means 23-a, 23-b, 23-c,..., 23-n are transferred one step (step S107). Then, as shown in FIG. 7, the charge during light emission and the charge during non-light emission are alternately accumulated in the horizontal charge transfer means 24.

At this point, the horizontal charge transfer means 24
Since the charges of the even-numbered lines have already been read out, all the charges accumulated in the horizontal direction charge transfer means 24 are shifted and extracted (step S11).
1). After the removal, the flag C is returned to "0" (step S112).

In this way, the vertical charge transfer means 23-a, 23-b, 23-c,..., 23-n shift the charge of each line to the horizontal charge transfer means 24 by one stage. In addition, the horizontal charge transfer means 24 comprises the vertical charge transfer means 23-a, 23-b, 23-c,
, 23-n, and shifts the group of electric charges in line units and outputs the resultant as an image signal.

The image signal of each line obtained from the horizontal charge transfer means 24 is an alternate arrangement of a light emitting image and a non-light emitting image for each pixel. Each pixel corresponds to an adjacent position when viewed from the image, and the distance difference is a slight difference. Thus, taking the difference between the adjacent pixels gives, in effect, an image of the difference between light emission and non-light emission. Would be equivalent.

The CCD area image sensor 21 used in the present system comprises vertical charge transfer means 23-a, 23
-B, 23-c,..., 23-n, that is, each vertical charge transfer means 23-a, 23-b, 23-c,
Since the element structure having the horizontal direction charge transfer means 24 having twice the number of pixels (double the number of pixels) that can be held by .about., 23-n is used, as described above, Charges for two columns, one line and one even line, can be alternately transferred and held in one horizontal charge transfer unit 24 in pixel units.

The difference between the light emission and non-light emission is calculated by the difference circuit 6 (step S113). The accumulation of the charge is performed until the charge accumulated in the vertical charge transfer means is exhausted (step S114).

That is, the vertical charge transfer means 23-
In a, 23-b, 23-c,..., and 23-n, the charge of the image of the odd-numbered line as the illuminated image and the charge of the image of the even-numbered line as the non-illuminated image are alternately arranged. Since the charges are held in a state, the charges of these lines are shifted one stage (for one pixel) to the horizontal charge transfer means 24 as the charge transfer means for outputting the image signal, and The charge transfer means 24 includes the vertical charge transfer means 23-a, 23-b, 23-c,.
-The charge group is shifted every time it is received by two stages from -n and output as an image signal.

In the manner described above, the reflection from the object other than the target can be accumulated in the image of the even-numbered line as non-emission charge. When the charge is subtracted from the image of the odd-numbered line, which is the charge at the time of light emission, purely reflected light from the target can be extracted. For example, reflected light as shown in FIG. 10 can be extracted. The reflected light from the portion near the input device is strong, and the intensity of the reflected light is inversely proportional to the square of the distance.

Therefore, this relationship can be expressed as follows. Rij = K / d ＾ 2 (Equation 1) Here, for example, when d = 0.5 m, the value of Rij is “25”.
The coefficient has been adjusted to be 5 ".

Therefore, the distance value can be obtained by solving Equation 1 for d. In other words, by extracting the reflected light,
The distance image can be extracted. After that, step S1
The process returns to step 01, and the acquisition of the reflected light is repeated. If this repetition is performed, for example, 30 times per second, 30 reflected images (distance images) per second can be extracted.

In this system, considering that the conventional reflection image (distance image) extraction by image processing is 1/6 or less per second, the system of the present invention is "30".
/ (1/6) = 180 ", which is a 180-fold increase in performance. Even if the system cost is simply compared,
It can be seen that the ratio can be reduced to 1/180 or less for "1/6" versus "30". In addition, since a CCD-type area image sensor is used to cope with the readout control, a realistic system is highly feasible and practical.

<Effects of First Embodiment> According to the first embodiment, a reflection image of a hand or the like can be obtained easily and in real time. This makes it unnecessary to cut out the image of the object, which is the most difficult in the conventional image processing and has been a barrier to the application of the image processing. Therefore, according to the present embodiment, it is possible to easily and stably provide various image processing which has been difficult to put into practical use at a low cost by using commercially available components. It will revolutionize a wide range of markets such as entertainment.

(Modification of First Embodiment) In the above-described first embodiment, the horizontal charge transfer means 24 shifts one stage at a time, and accumulates charges during light emission and during non-light emission in a line. Then, the signal is sent to the difference circuit. However, it is not necessarily limited to this.

For example, when the horizontal charge transfer means 24 is
It is also possible to use a CCD area image sensor having two stages (two sets). In the case of this two-stage CCD area image sensor, for example, first, electric charges in a non-light emitting state are arranged in the upper horizontal charge transfer means 24.

Then, in the next transfer, the non-emission charge is transferred to the lower horizontal charge transfer means 24, and then the vertical charge transfer means 23-a, 23-b,. n
As shown in FIG. 11, the charge at the time of light emission is transferred from the horizontal charge transfer means 24a and 24 in the upper and lower stages.
b. Charges at the time of light emission and at the time of non-light emission are separately arranged. This may be subtracted by the difference circuit 6.

Alternatively, the horizontal charge transfer means 24 transfers the charges in one stage in units of lines, first transfers the non-light-emitting charges for one line to the difference circuit 6, and then transfers the light-emitting charges. It is also possible to adopt a method of transferring.

Alternatively, a delay element for one line is provided at the output of the CCD area imaging sensor, and the output of the CCD area imaging sensor is set to 1 when no light is emitted.
If the output of the column is input to this delay element and one column at the time of the next light emission is taken out without passing through the delay element, the output at the time of non-light emission and the time of light emission can be obtained at the same time. This can be input to the difference circuit 6 to obtain a difference output.

In FIG. 16, the non-emission charge that has entered the horizontal CCD is extracted and input to the 1H delay line. The 1H delay line provides a time delay of 1H (one horizontal scan) between signal input and output. In FIG. 17, the charge at the time of light emission that has entered the horizontal CCD is extracted and then directly input to the difference circuit.

Therefore, as shown in FIG. 18, the non-emission signal passing through the 1H delay line and the emission signal directly input to the difference circuit are simultaneously input to the difference circuit. There is no need to have a buffer or the like. Since a typical CCD image sensor has only one horizontal CCD, adopting this configuration makes it easier to realize. Whether the signal output from the CCD is sent to the delay line or directly to the difference circuit is determined by a switch (not shown), which is controlled by a timing generator. Note that a signal at the time of light emission may be input to the delay line, and a signal at the time of non-light emission may be directly input to the difference circuit.

In the first embodiment, the even-numbered lines and the odd-numbered lines are assigned for image extraction at the time of light emission and at the time of non-light emission. However, the present invention is not limited to this. Further, in order to increase the accuracy, it is also possible to use twice as many lines as the lines for storing the charges during non-light emission in order to store the charges during light emission.

(Second Embodiment) The schematic configuration of the second embodiment is similar to that of the embodiment. However, in this example, the light receiving element and the vertical charge transfer means 23-a, 23-b, 23-
This is an example of implementation in the case of using another CCD area image sensor having a different charge transfer method between n.

In the first embodiment, among the control signals generated by the timing generator 5, G1 is an odd line,
G2 opens and closes the gate of charge transfer from the light receiving element 22 of the even-numbered line to the vertical charge transfer means 23.
Then, the charges converted / stored by the respective light receiving elements 22 have been transferred to the corresponding positions of the corresponding vertical charge transfer means 23 of the light receiving elements 22.

On the other hand, in the second embodiment, the light receiving element 22 has two adjacent light lines regardless of the even line or the odd line.
An area image sensor having a gate structure is used in which the charges converted / stored by the light receiving elements 22 forming one pair are transferred to the same vertical charge transfer means 23.

FIG. 12 shows an operation example of charge accumulation in such an area image sensor, and FIG. 13 shows a flow chart of processing. The flowchart of FIG. 13 is different from the flowchart of FIG. 8 in that the process of step S204 is added. That is, in step S202, a voltage is applied to G1 to open the gate, and the light receiving elements 22-a1, 22 of the odd-numbered lines are opened.
-A3, ~, 22-an, ~, 22-n1, 22-n
3, ..., 22-ni to vertical charge transfer means 23
a, 23b, to 23n.

At this point, charges are stored in the vertical charge transfer means 23a, 23b,..., 23n as in FIG. Each light receiving element (PD) 22-a1,
After resetting 22-a2, ..., 22-n1, ..., 22-ni (step S203), in step S204, the vertical charge transfer means 23-a, 23-b, ..., 23-n
Shifts the electric charge stored in the memory cell by one step. Then, the vertical charge transfer means 23-a, 23-b,.
The charge stored and held in 23-n is in a state as shown in FIG. That is, the odd-numbered line positions become empty, and the electric charges are located at the even-numbered line positions.

Next, the light emitting element 3 is turned off (step S205), a voltage is applied to G2, and the gate is opened.
The received charges are transferred from the even-numbered lines to the vertical charge transfer units 23-a, 23-b, 23-c,..., 23-n (step S206). At this time, the gate is the same as the vertical charge transfer means 23-a, 23-b, 23-c,.
-N.

As a result, the vertical charge transfer means 23-a, 23-b, 23-c,.
As shown in FIG. Thereafter, as in the example of the first embodiment, the reflected image (distance image) can be acquired by transferring the image to the horizontal charge transfer unit 24 and calculating the difference.

<Effects of Second Embodiment> According to the second embodiment described above, light is received by the same light emitting element (PD) during light emission and during non-light emission. A highly accurate reflection image (distance image) can be obtained without being affected by the above, and the effect is great.

(Modification of Second Embodiment) As for the configuration of the charge transfer means 24 for the horizontal direction, a plurality of configurations can be adopted as in the first embodiment.

In the second embodiment, the light receiving elements 22-a1, 22-a3,..., 22-an,.
Only 22-n1, 22-n3, ..., 22-ni are used. Thus, the light receiving elements 22-a1, 22-2 of the odd lines
2-a3, ~, 22-an, ~, 22-n1, 22-n
The near-infrared wavelength band-pass filter is provided on the light-receiving surface side of 3, to 22-ni, and the light emitted by the luminous body is
Odd line light receiving elements 22-a1, 22-a3,.
2-an, ~, 22-n1, 22-n3, ~, 22-n
Only i can receive light.

Then, the light receiving elements 22 of the remaining even-numbered lines
-A2,22-a4, ~, 22-an-1, ~, 22-
n2, 22-n4,..., 22-ni-1 have a configuration in which a color filter is provided on the light receiving surface side, so that the light receiving elements 22-a2, 22-a4,.
1, ~, 22-n2, 22-n4, ~, 22-ni-1
Is used, a normal color imaging can be performed. Therefore, by selectively using the odd-numbered lines and the even-numbered lines, it is possible to obtain both a color image and a distance image with a single input device.

Thus, a chroma key camera or the like can be realized, and the effect is great. Although various embodiments have been described above, in short, the present invention provides a method for obtaining a distance image.
In order to obtain a light reflection image of a subject from a difference image of the subject image between when the light is irradiated and when the light is not irradiated, a light emitting unit that irradiates the light to the subject and an imaging unit in which a plurality of light receiving elements that perform photoelectric conversion are two-dimensionally arranged. And an area image sensor having CCD type charge transfer means for transferring and taking out the charge obtained by these image pickup units; and driving the CCD type charge transfer means of the area image sensor and transferring the light from the light receiving element to the CCD type charge transfer means. The charge transfer timing is controlled so that the amount of charge received when the light emitting unit emits light and the amount of charge received when the light emitting unit does not emit light in all or each CCD type charge transfer unit are controlled. And a timing signal generating means for controlling so as to be arranged in a predetermined arrangement.

According to such a configuration, the pixels of the subject image at the time of light emission and at the time of non-light emission are alternately arranged in pixel units by timing control from the two-dimensionally arranged light receiving elements of the CCD area image sensor. Directly connect the image to the CC
It can be obtained from a D-type area image sensor, and a difference image can be obtained in real time by extracting a difference between pixels, so that a reflection image of a hand or the like can be obtained easily and in real time. . This makes it unnecessary to cut out the image of the object, which is the most difficult in the conventional image processing and has been a barrier to the application of the image processing.

Therefore, according to the present invention, it is possible to provide various types of image processing which have been difficult to put into practical use by using commercially available parts, easily, stably, and at low cost. It will revolutionize a wide range of markets such as home / entertainment. As a result, pointing and change of the viewpoint in the three-dimensional space can be easily performed, and a natural motion can be given to an animation character or the like using a gesture or motion of the user as it is.

C in the CCD type area image sensor
In the CD-type charge transfer means, the charge at the time of light emission and the charge at the time of non-light emission are arranged in a predetermined arrangement. It is a difference circuit that acquires this difference and extracts only the reflected light of the light emitting means. . Since a typical CCD area image sensor does not have a difference circuit, it is typical to provide a difference circuit after the output of the CCD image sensor.

However, of course, the CC having the built-in difference circuit
A D-type image sensor can also be used. The pattern of the output signal differs depending on the configuration of the CCD type charge transfer means of the CCD type area image sensor. As shown in FIG. 7, when two corresponding outputs (that is, the difference operation is performed) appear alternately, the difference circuit is simple. The first signal may be clamped by a sample hold circuit or the like, and the difference operation may be performed when the next signal is input. The difference operation can be easily realized by, for example, a differential amplifier. As shown in FIG. 11, even when two corresponding signals are output simultaneously, a difference output can be easily obtained.

As shown in FIG. 18, the case where two corresponding signals are output alternately in line units is slightly complicated.
As shown in FIG. 18, if a signal output earlier is delayed using a delay line, two corresponding signals are simultaneously input to the difference circuit. There is also a method of preparing a buffer for one line in the difference circuit.

Although the present embodiment mainly deals with a configuration in which a difference is converted before conversion into a digital signal by an AD converter, the present invention is not limited to this. For example, the difference can be calculated after the signal is converted into a digital signal by an AD converter. However, in this case, there is a demerit that the dynamic range is sacrificed by the difference calculation. For example, consider a case where the output during non-light emission is about three-quarters of the output during light emission. And this together 10
When the difference is calculated after digitization by a bit AD converter, the dynamic range is reduced by 2 bits.

Such a phenomenon tends to occur particularly when the output when light is not emitted is relatively large. In such a case, there is a great merit that the difference can be calculated before AD conversion. Conversely, if the difference is performed after the A / D conversion, for example, if one image buffer is secured, as described in the present embodiment, a special charge is stored in the CCD type charge transfer means of the area image sensor. This can be realized without arranging charges in an array. However, in addition to the decrease in the dynamic range described above, there are disadvantages such as the necessity of preparing a large image buffer for one image and the increase in computational cost after digitization.

Also, for each frame or field,
An image at the time of light emission and an image at the time of non-light emission are sequentially output, and there is a time interval of about 1/60 second between the two imagings.
When capturing a fast motion, image blur occurs.
The method according to the present embodiment does not have such a problem because both an image during light emission and an image during no light emission can be captured in a very short time. It should be noted that the present invention is not limited to the fulfilled embodiments, and can be implemented with appropriate modifications.

[0110]

As described in detail above, according to the present invention,
Provide a practical three-dimensional information input device that can easily perform pointing and change of viewpoint in a three-dimensional space, and make natural movements to animated characters and the like using the user's gestures and movements as they are. can do.

[Brief description of the drawings]

FIG. 1 is a diagram for explaining the present invention, and is a diagram for explaining a configuration example of a first embodiment of the present invention.

FIG. 2 is a diagram for explaining the present invention, and is a diagram for explaining a more specific configuration example of the first embodiment of the present invention.

FIG. 3 is a diagram for explaining the present invention, and is a diagram for explaining a more specific configuration example of the first embodiment of the present invention.

FIG. 4 is a view for explaining the first embodiment of the present invention.

FIG. 5 is a diagram for explaining the first embodiment of the present invention.

FIG. 6 is a view for explaining the first embodiment of the present invention.

FIG. 7 is a diagram for explaining the present invention and is a diagram for explaining an operation example of charge transfer in the first embodiment.

FIG. 8 is a diagram for explaining the present invention, showing an example of a processing flow in the first embodiment.

FIG. 9 is a diagram for explaining the present invention, showing an example of a signal for controlling charge transfer and an operation transition in the first embodiment.

FIG. 10 is a diagram for explaining the present invention, and is a diagram showing an example of a reflection image acquired in the first embodiment.

FIG. 11 is a diagram for explaining the present invention, and is a diagram showing an operation example of charge transfer in a modification of the first embodiment.

FIG. 12 is a view for explaining the first embodiment of the present invention.

FIG. 13 is a view for explaining the present invention, showing an example of a processing flow in the second embodiment;

FIG. 14 is a diagram for explaining the present invention, showing an operation example of charge transfer in the second embodiment.

FIG. 15 is a view for explaining a conventional example.

FIG. 16 is a diagram for explaining the present invention, and is a diagram showing an operation example of charge transfer in a modification of the first embodiment.

FIG. 17 is a diagram for explaining the present invention, showing an operation example of charge transfer in a modification of the first embodiment.

FIG. 18 is a view for explaining the present invention, showing an operation example of charge transfer in a modification of the first embodiment.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Image pick-up device 2 ... Light emission control means 3 ... LED (light emitting element) as a light source 4 ... Optical system 5 ... Timing generator 6 ... Difference circuit 7 ... Analog amplifier 8 ... Analog / digital converter (A / D converter) 9 ... System control unit 10 ... Frame buffer 11 ... Interface circuit to personal computer 21 ... CCD area image sensor 22 (22-a1, ..., 22-ni) ... Light receiving element 23 (23-a, ..., 23-n) ... Vertical charge transfer means 24... Horizontal charge transfer means (image signal output charge transfer means).

Claims (9)

[Claims] 1. An apparatus for obtaining a difference image between a subject image at the time of light irradiation and a subject image at the time of no light irradiation, wherein a light emitting means for irradiating light to a subject and a plurality of light receiving elements for photoelectric conversion are two-dimensionally arranged. An image pickup unit and C for transferring and taking out charges obtained by these image pickup units
An area image sensor having a CD-type charge transfer means; and controlling the timing of driving the CCD-type charge transfer means of the area image sensor and transferring the charge from the light receiving element to the CCD-type charge transfer means. Or, in each CCD type charge transfer means, there is provided timing signal generating means for controlling the amount of charge received when the light emitting means emits light and the amount of charge received when the light emitting means does not emit light in a predetermined arrangement. An information input device, characterized in that: 2. The information according to claim 1, wherein the timing signal generating means has a function of controlling the transfer of electric charges from only a part of the light receiving elements of the area image sensor to the CCD type electric charge transferring means. Input device. 3. The timing signal generating means transfers from only one of the light receiving elements at the even line position and the light receiving element at the odd line position among the two-dimensionally arranged light receiving elements of the area image sensor in synchronization with the light emitting means. 2. The information input device according to claim 1, further comprising a function of generating a signal for performing the operation. 4. An area image sensor passes light emitted from said light emitting means only to a light receiving element provided for imaging in synchronization with the light emitting means among light receiving elements arranged two-dimensionally, and transmits unnecessary light. 2. The information input device according to claim 1, wherein a filter that does not pass is added. 5. A CCD in the CCD type area image sensor in synchronization with a signal from a timing signal generating means.
2. The information input device according to claim 1, further comprising: a charge transfer control signal generation unit that generates a control signal for performing transfer control in the D charge transfer unit. 6. The information input device according to claim 5, wherein the charge transfer control signal generating means generates a signal for transferring charges when the light emitting means does not emit light. . 7. A charge transfer control signal generating means for alternately transferring, from among the two-dimensionally arranged light receiving elements of the area image sensor, even line positions and odd line positions.
2. The information input device according to claim 1, wherein a signal for transfer control is generated in synchronization with the light emitting means. 8. An information input device, comprising: a difference circuit for extracting a difference between a light-receiving charge amount during light emission and a light-receiving charge amount during non-light emission output from the area image sensor. 9. An apparatus for obtaining a difference image between a subject image at the time of light irradiation and a subject image at the time of no light irradiation. An image pickup unit and C for transferring and taking out charges obtained by these image pickup units
An area image sensor having a CD-type charge transfer means; controlling the drive of the CCD-type charge transfer means of the area image sensor and the timing of charge transfer from the light receiving element to the CCD-type charge transfer means; Or, in each CCD type charge transfer means, a timing signal generating means for controlling the amount of charge received when the light emitting means emits light and the amount of charge received when the light emitting means does not emit light in a predetermined arrangement; A delay line for delaying the amount of charge received from the area image sensor during light emission or non-light emission by one horizontal scanning time based on the timing signal of the signal generating means; Difference between the charge amount at the time of light emission and the charge amount at the time of non-emission output directly from the Information input device, characterized in that it comprises a differential circuit for outputting a difference between the charge amount of the non-emission time which is input to the ray lines and the charge amount at the time of light emission output directly from the area image sensor.