JP2002007030A - Movement detecting device and operation input device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an operation input device capable of accurately recognizing the spatial position, attitude and movement of an operator's hand. SOLUTION: This operation input device has hand back detecting means (12X, 12Y, 12Z, 14X, 14Y and 14Z) which are attached to the back of a hand of an operator and detect the movement or attitude of the back of the operator' s hand, thumb and finger attitude detecting means (7I, 7TY and 7TX) which are attached to operator's thumb and finger and detect the attitude of the operator's thumb and finger, a hand shape estimating means for calculating the entire shape of the operator's hand on the basis of the outputs of the hand back detecting means and the thumb and finger attitude detecting means, an operation input analyzing means for generating a prescribed command by using the output of the hand shape estimating means, and image processing chips 15X, 15Y and 15Z provided with an imaging means for picking up the surroundings, a comparing means for comparing images picked up by the imaging means at different times and an acquiring means for acquiring hand movement information on the basis of the comparison results of the comparing means.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a motion detection device and an operation input device.

[0002]

2. Description of the Related Art There has been known a technique in which a sensor is attached to an operator's hand to detect the shape and movement of the hand and generate a signal based on the detection result.

For example, US Pat. No. 5,097,252 discloses a technique in which a plurality of sensors each having a light source and an optical sensor connected by a light guide path such as an optical fiber are attached to a joint of a hand to detect the bending of the joint. ing.

[0004] Japanese Patent Application Laid-Open No. 9-62437 discloses that
A mouse substitute for detecting two-dimensional movement of the hand by arranging two acceleration sensors on the back of a gloved hand and arranging one strain gauge at a joint of the index finger of the hand to detect bending and stretching movement of the index finger; Is disclosed.

Also, a patent application filed by the present applicant (Japanese Patent Application No.
No. 0-302236), a three-axis angular velocity sensor and a three-axis acceleration sensor for detecting a position and a posture are arranged on the back of the hand, and a finger is provided at the end of the index finger, the end of the middle finger, the end of the thumb and the center. An operation input device that disposes a one-axis angular velocity sensor that detects bending, estimates a hand shape from the position and posture of the back of the hand and the posture of the finger, and generates a command signal based on the shape of the hand is disclosed. I have.

Further, the present inventor has filed a patent application (Japanese Patent Application No.
No. 307076), the acceleration accompanying the movement is sensed by a sensor attached to the body of the operator so that the operator can use the operation input device even while riding on a moving body such as a car. A technique for correcting the effect of the above is disclosed.

[0007]

SUMMARY OF THE INVENTION The method described in Japanese Patent Application No. 10-302236 or Japanese Patent Application No. 11-307076 by the present inventor, that is, the position and posture of the back of the operator's hand are set to three.
Angular velocity sensor of three axes and acceleration sensor of three axes, the posture of the finger is detected by the angular velocity of one axis at the end, and the shape of the hand is estimated based on these data. A method for generating a signal is described in US Pat.
Compared to the method of arranging sensors at joints as disclosed in JP-A-52-52 and JP-A-9-62437, there is no need to determine the sensor position in consideration of the size of an individual's hand, and it is versatile. An operation input device that can be easily used by anyone is provided.

However, Japanese Patent Application No. 10-302236.
In Japanese Patent Application No. Hei 11-307076, the acceleration sensor is used to detect the translational motion of the back of the operator's hand, such as speed and position, by integrating the time. It is difficult to distinguish when moving fast,
There is a problem in recognizing a gesture in a state where the hand is stopped and a gesture in which the hand is moved at a substantially constant speed.

The present invention has been made in view of such a problem, and an object of the present invention is to provide a motion detecting device capable of accurately recognizing a spatial position, a posture, and a motion of an object to be mounted.

Another object of the present invention is to provide an operation input device capable of accurately recognizing the spatial position, posture and movement of the operator's hand.

[0011]

According to a first aspect of the present invention, there is provided a motion detecting apparatus for detecting the motion of an object using at least one of an acceleration sensor and an angular velocity sensor. An imaging unit that captures an image of the surroundings; a comparison unit that compares images captured at different times by the imaging unit; and an acquisition unit that acquires motion information of the mounted object based on a comparison result of the comparison unit. Have.

According to a second aspect of the present invention, there is provided a hand back detecting means which is attached to the back of the hand of an operator and detects at least one of an acceleration and an angular velocity applied to the hand, and a predetermined hand using the detection result of the hand back detecting means. An operation input device having operation input analysis means for generating a command in the method of (1), an imaging means for imaging the surroundings, a comparison means for comparing images at different time points taken by the imaging means, and a comparison between the comparison means Acquiring means for acquiring hand movement information based on the result.

According to a third aspect of the present invention, there is provided a back detection means mounted on the back of an operator's hand to detect the movement or posture of the back of the operator's hand, and mounted on a finger of the operator, Finger posture detecting means for detecting the posture of the hand, hand shape estimating means for calculating the shape of the entire hand of the operator based on the outputs of the back of hand detecting means and the finger posture detecting means, and an output of the hand shape estimating means. An operation input device having operation input analysis means for generating a predetermined command using
The apparatus further includes imaging means for imaging the surroundings, comparison means for comparing images taken at different points in time taken by the imaging means, and acquisition means for acquiring movement information of the hand based on the comparison result of the comparison means.

[0014]

DESCRIPTION OF THE PREFERRED EMBODIMENTS First, an outline of the present embodiment will be described. In order to detect the position and posture of the operator's hand and its movement, a hand sensor including a three-axis acceleration sensor and a three-axis angular velocity sensor is mounted. Therefore, the rotational motion and the translational motion of the back can be obtained from the information obtained from the acceleration sensor and the angular velocity sensor. However, since the acceleration sensor combines the gravitational acceleration due to gravity and the inertial acceleration due to inertial motion, the inclination component, which is the gravitational acceleration, is a low-pass filter, and angle information obtained by time-integrating the angular velocity of the angular velocity sensor is used. To separate. The acceleration information based on the inertial motion output from the acceleration sensor obtained in this manner indicates that the inertial acceleration becomes zero when the acceleration is lost, even if the vehicle is moving at a constant speed or stopped. It doesn't work. Therefore, in the present embodiment, in addition to the back sensor for detecting the position and posture of the back of the hand,
An image space sensor capable of detecting motion information in two orthogonal directions is added to capture an image around the wearer. The relative motion between the image sensor and the surrounding object can be known from the motion of the image input to the image space sensor. The image signal obtained from the image space sensor includes motion change information indicating whether or not the subject has moved, and moving direction information indicating in which direction the subject has moved. Here, utilizing this fact, the amount of movement change in the orthogonal direction on the image is detected based on the change in the feature points between the two preceding and succeeding frames of the surrounding image input in time series to the image space sensor. . Here, the focal length of the image space sensor is set to almost infinity. Thus, the information from the acceleration sensor and the angular velocity sensor
By combining the motion information from the image space sensor, the translational motion of the back can be accurately obtained.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. FIG. 1 is a diagram showing a configuration example of an operation input device in which an image processing chip on which the above-mentioned image space sensor is mounted is arranged. The image processing chip includes, in addition to an image space sensor for imaging, a comparison unit that compares the captured images at different time points, and an acquisition unit that acquires motion information of the image based on a comparison result of the comparison unit. Shall be provided. As an image processing chip, for example, HDNS-200 manufactured by Hewlett-Packard (HP) Co., Ltd.
0 can be used. Although this image processing chip is a chip dedicated to short distances, it is easy to apply an optical adapter for long distances or to improve the logic circuit and apply it to long distance imaging.

As shown in FIG. 1, Y in the fingertip coordinate system XYZ
Uniaxial angular velocity sensors 7I, 7TY, 7T as finger posture detecting means for detecting angle information about the axis (Pitch direction)
X are placed at the end of the index finger of the right hand, the end of the thumb and the center of the thumb, respectively, and the position of the back of the right hand (Xb, Y
b, Zb) and posture (Pitch, Roll, Yaw) as acceleration detecting means 12X, 1
2Y, 12Z and angular velocity sensors 14X, 14Y, 14Z
On the back of the right hand.

Here, the acceleration sensors 12X, 12Y, 12
Z is constituted by a semiconductor type acceleration sensor, and here is constituted by combining two-axis type acceleration sensors 12X and 12Y and a one-axis type acceleration sensor 12Z. Further, as the angular velocity sensors 14X, 14Y, and 14Z, piezoelectric vibrating gyro sensors that detect angular velocity momentum in the rotation direction of one axis are used.

Further, the acceleration sensors 12X, 12Y, 12Z and the angular velocity sensors 14X, 14X,
14Y, 14Z, the image processing chips 15X, 1
5Y and 15Z are respectively arranged along three substantially orthogonal backspace coordinate axes X, Y and Z.

FIG. 2 shows the image processing chips 15X, 15Y, 15Z arranged on the respective mounting axes X, Y, Z and the detection directions thereof. As the output signal of the image processing chip alone, the displacement output in the horizontal and vertical directions when the black circle position is set to the upper left is taken as the image translation movement information Ix, Iy. The correspondence in the output coordinate axis direction when mounted in three spatial directions as shown in the figure is shown. In the image processing chip 15X, motion information in the y direction and the z direction is obtained. The image processing chip 15Y can obtain motion information in the x direction and the z direction. The image processing chip 15Z can obtain motion information in the x and y directions. In such an arrangement of the image processing chips, if the back of the hand independently translates in the + X direction, the + Y direction, and the + Z direction, each of the image processing chips 15X,
From 15Y and 15Z, a combination of motion detection information as shown in FIG. 3 is obtained. Each of the image processing chips 15X, 15Y, and 15Z when the back of the hand rotates by + α around the X axis, by + β around the Y axis, or by + γ around the Z axis. Can obtain the motion detection information of the combination in the output coordinate axis direction as shown in FIG. By arranging the image processing chips along the three-axis directions in this way, it is possible to obtain posture information of the six-axis space of the back of the hand by only image processing.

Further, it is possible to determine whether the back of the hand is in a constant acceleration state or a stopped state based on such motion detection information, and to combine this with the output information of the angular velocity sensor and the acceleration sensor. Thus, the posture of the back of the hand can be accurately obtained.

FIG. 4 is a diagram showing a modification of the configuration shown in FIG. 1, in which one image processing chip (15X is shown in the figure)
It is characterized by only being placed on the back of the hand. In this case, the image processing chip is used to acquire translation information on the translation of the back of the hand in the operation mode when a predetermined operation mode such as a pointing mode is entered by a hand gesture. For example, when entering the virtual mouse mode, in order to enter a mode in which translational movement is performed without performing rotation, it is necessary to perform arithmetic processing assuming that the back is only performing translational movement in the up, down, left, and right directions Is possible. Also, the image processing chip is replaced with the 15Z of FIG.
In the case of using only one, it is possible to acquire the translational motion of front and rear, left and right, and it is possible to recognize a gesture of operating the back of the hand on a plane such as a desk. Thus, it is assumed that the image processing chips are arranged in a form corresponding to the operation mode.

Note that the detection output information when only one image processing chip is arranged is not used for the posture detection of the back of the hand.

FIG. 5 is a diagram for explaining the relationship and flow of each sensor information in the motion detection operation of the operation input device according to the present embodiment. This figure shows a signal measurement routine for motion detection operation and a measurement calculation routine for arithmetically processing the signal, which is periodically and repeatedly processed by an internal timer.

First, when signal detection is started, measurement of the acceleration signal Ah (d) accompanying the movement of the back of the hand by the acceleration sensors 12X, 12Y and 12Z and the angular velocity sensor 14
Measurement of the angular velocity signal Gh (d) with respect to the rotation axis of the back by X, 14Y, 14Z, and image processing chips 15X, 15
Y, 15Z image translation movement information Ix (d), Iy
(D) Measurement and one-axis angular velocity sensors 7I, 7T of each fingertip
Measurement of the fingertip angular velocity signal G (i) by Y, 7TX is performed in parallel (steps S1, S12, S18, S2).
3). d is a number for identifying three sensors in the order of the X, Y, and Z axes, and d = 1, 2, and 3. Fingertip angular velocity signal G (i)
I is a number for identifying the angular velocity sensor of each fingertip.

After the measurement of the acceleration signal Ah (d) in step S1, in step S2 the back of the hand is moved depending on whether or not the output signal has changed by comparing the previous signal data of the acceleration sensor itself. It is determined whether it is moving, stopped or moving at a constant speed. It is determined whether the absolute value of the signal difference is equal to or greater than a certain threshold. If there is a change in the output signal for each of the three axis sensor signals, an acceleration stop flag Astop is determined in step S3.
0 is written in (d), and if there is no change in the output signal, 1 is set in the acceleration stop flag Astop (d) in step S4.
Write.

Also in step S12, the angular velocity signal Gh
After the measurement of (d), the process proceeds to step S13, where the hand back angular velocity signal is compared with the previous signal data to determine whether or not the value is within a predetermined range. If it is within the predetermined range, it is determined that there is no rotational movement, and the routine proceeds to step S15, where 1 is written into the angular velocity stop flag Gstop (d). If it is not within the predetermined range, the routine proceeds to step S14. Write 0 to the angular velocity stop flag Gstop (d).

After the measurement of the image translation movement in step S18, the process proceeds to step S19, where the image processing chip 15
It is determined whether or not there is a relative motion between the back of hand and the peripheral image based on whether or not the outputs of X, 15Y, and 15Z have been displaced. If the signals do not displace for each of the three axes, step S21 is performed.
And writes 1 in the image stop flag Istop (d).
If the displacement has occurred, the process proceeds to step S20, where 0 is written to the image stop flag Istop (d).

In step S16, a signal obtained by moving and averaging the angular velocity signal measured in step S12 and the angular velocity signal at the time of stop determination is used as an offset signal point.
The angular velocity ω (α, β, γ) in the shaft rotation direction is calculated.

In step S11, the inclination posture (θx, θy) of the back of the hand with respect to the gravity vector axis is calculated from the acceleration information measured in step S1.

Then, in step S17, step S
16 is temporally integrated with the immediately preceding back posture θ to obtain a temporary back posture θ ′, and the inclination back posture information (θ) calculated in step S11 is obtained.
(x, θy) is used as correction data, and a drift correction calculation is performed to calculate a new back posture angle θ (α, β, γ).

Although not shown here, the internal information is initialized when the power is turned on or when a reset signal is executed. The initial value of the acceleration signal Ah (d) is held, and the square root of the sum of squares (G
0) is held. The back angle posture θ (α, β, γ)
Α = θy, β
= Θx, γ = 0. In addition, each angular velocity signal Gh
(D) and G (i) are held as initial offset signals. Each stop determination flag is also initialized to 1. At this time, the back of the hand may be inclined, but it is necessary to satisfy the stopping condition. However, that time is also instantaneous.

In step S5, based on the angular velocity information calculated in step S16 to remove the gravitational acceleration component from the acceleration signal measured in step S1, and the sensor signal value (G0) of the gravitational vector 1G held at the time of initialization. The translational acceleration component α (d) is extracted. Next, step S6
To determine the movement of the back of the hand. Details of the motion determination processing will be described later.

In step S22, a process for determining an operation direction is performed based on the conditions shown in FIG. 3 based on the image translational motion information measured in step S18.

After the start of the measurement of the fingertip angular velocity signal in step S23, the process proceeds to step S24, and the process proceeds to step S23.
The offset processing and the scale conversion processing are performed from the fingertip angular velocity signal measured in step S15, and the processing of the finger joint angular velocity Fω (i) of the finger tip with respect to the back of the hand is performed based on the backside angular velocity information ω calculated in step S16. Is used to calculate the fingertip angle Fθ (i).

The details of the motion determination process in step S6 will be described below with reference to FIG. First, in step S100, it is checked whether the angular speed stop flag Gstop (d) is 0 or 1, and if it is 0, there is an output from the gyro sensors which are the angular speed sensors 14X, 14Y, and 14Z. It is determined that the back is rotating and 0 is written in the back stop flag Hstop (step S10).
5) Return. Also, all the angular velocity stop flags G
If stop (d) is 1, step S101
To check whether the acceleration stop flag Astop (d) is 0 or 1.

If any of the acceleration stop flags Astop (d) is 0, the acceleration sensors 12X, 12Y,
Since there is a displacement output of 12Z, it is determined that the back is in translation, and 0 is written in the back stop flag Hstop (step S105), and the routine returns. If the acceleration stop flag Astop (d) is set to 1, step S10
The process proceeds to 2 to check whether the image stop flag Istop (d) is 0 or 1.

When the image stop flag Istop (d) is 0, it means that there is a relative movement between the back of the hand of the image space sensor and the object in the peripheral image. Therefore, the translation operation flag Pmove is used to determine whether the back of the hand has moved or the hand has stopped and the surrounding object has moved.
Check (d).

If the translation operation flag Pmove (d) = 1 in step S103, it is determined that the back of the hand is exercising because the translation operation is being performed in the previous measurement calculation routine, and 0 is written in the hand back stop flag Hstop. (Step S10
5) Return. It is presumed that the movement of the back of the hand at this time does not rotate and the translational movement is moving at a constant speed.

If the translation operation flag Pmove (d) = 0 in step S103, the translation operation has not been performed in the previous measurement operation routine, so that not the constant speed translation of the back of the hand but the movement of the surrounding object has occurred. It is determined to be due to image noise, and it is determined that the back is stopped, and the back stop flag Hst
Write 1 to op (step S105) and return.

If the image stop flag Istop (d) is 1 in step S102, it is determined that the back of the hand is completely stopped because there is no translational movement output from the image space sensor, and 1 is set to the back of hand back flag Hstop. Write (step S105) and return.

Here, returning to the flow of FIG. 5, step S6 is performed.
After the motion determination process, the process proceeds to step S7 to check whether the backrest stop flag Hstop is 0 or 1. If it is 1, the process proceeds to step S10 to stop the translation calculation. Translation operation flag Pmove
Is written to the CPU to perform calculations for various sensor processes under the complete stop condition, and then returns from the measurement calculation routine.

When the back stop flag Hstop is 0, the process proceeds to step S8 to write 1 in the translation operation flag Pmove (d) in order to start the translation operation.
In the next step S9, the translational acceleration is time-integrated based on the translational acceleration component α (d) in step S5 in consideration of the determination result of the operation direction in step S22, and the translational velocity v (d) and the translational position are calculated. Information l (d) is obtained.

According to the above-described embodiment, it is possible to accurately distinguish between the stop of the hand of the operator and the movement at a constant speed.

Further, in the above embodiment, the information of the image sensor is used to correct the calculation processing of the translation information by the acceleration sensor, but the drift correction processing and the angle calculation processing of the angular velocity sensor are performed in the same manner. Can be used.

In addition, in addition to the configuration of the above-described embodiment,
The hand shape can be estimated by using the hand shape estimation unit and the operation input analysis unit disclosed in Japanese Patent Application No. 10-302236 by the present applicant.

(Supplementary Note) The invention having the following configuration can be extracted from the above-described specific embodiments.

1. In a motion detection device that detects the motion of the mounted object using at least one of an acceleration sensor and an angular velocity sensor, an imaging unit that captures an image of a surrounding, and a comparison unit that compares images at different time points captured by the imaging unit, An exerciser for acquiring exercise information of the object to be worn based on a comparison result of the comparing means.

(Effect) It is possible to provide a motion detecting device capable of accurately distinguishing between the stop of the mounted object and the constant speed movement.

2. A hand back detection unit that is attached to the back of the operator's hand and detects at least one of acceleration and angular velocity applied to the hand, and an operation input analysis unit that generates a command in a predetermined method using a detection result of the back hand detection unit. An operation input device comprising: an imaging unit that images the surroundings; a comparison unit that compares images captured at different times by the imaging unit; and an acquisition unit that obtains hand movement information based on a comparison result of the comparison unit. And an operation input device.

(Effect) It is possible to provide an operation input device that can accurately recognize the spatial position, posture, and movement of the operator's hand.

3. Hand back detection means mounted on the back of the operator's hand and detecting the movement or posture of the back of the operator's hand, finger posture detection means mounted on the operator's finger and detecting the posture of the operator's finger, Hand shape estimating means for obtaining the shape of the entire hand of the operator based on the outputs of the back of hand detecting means and the finger posture detecting means, and an operation of generating a predetermined command using the output of the hand shape estimating means An operation input device having input analysis means, an imaging means for imaging the surroundings, a comparison means for comparing images at different time points taken by the imaging means, and a hand movement based on a comparison result of the comparison means. An operation input device, further comprising: an obtaining unit configured to obtain information.

(Effect) It is possible to provide an operation input device capable of accurately recognizing the spatial position, posture and movement of the operator's hand.

4. The imaging unit, the comparison unit, and the acquisition unit are realized on one chip. The motion detection device according to claim 1.

(Effects) In addition to the effects described above, a more compact motion detection device can be provided.

5. 1. The imaging unit, the comparison unit, and the acquisition unit are realized on one chip. Or 3. An operation input device according to item 1.

(Effect) Further, it is possible to provide an operation input device in which the sensor is not bulky.

6. The chip is mounted on the back of the hand, and has an operation mode in which the output of the chip is reflected in the output of the back detection means and an operation mode in which the output of the chip is not reflected in the output of the back detection means. 5. Operation input device as described.

(Effect) Information can be used for imaging only in a predetermined operation that requires a distinction between a stop of the back of the hand and a constant speed motion, and logic can be easily created.

7. 4. The three chips are mounted, and each chip captures images in three directions substantially orthogonal to each other. Or 6. An operation input device according to item 1.

(Effects) Information on six axes in space can be obtained only from an image.

[0061]

According to the first aspect of the present invention, it is possible to provide a motion detecting device capable of accurately recognizing a spatial position, a posture, and a motion of an object to be mounted.

According to the second or third aspect of the present invention, it is possible to provide an operation input device capable of accurately recognizing the spatial position, posture and movement of the operator's hand.

[Brief description of the drawings]

FIG. 1 is an image processing chip 15 equipped with an image space sensor.
It is a figure showing the example of composition of the operation input device which arranged X, 15Y, and 15Z.

FIG. 2 is a diagram illustrating image processing chips 15X, 15Y, and 15Z arranged on respective mounting axes X, Y, and Z, and detection directions thereof.

FIG. 3 is a diagram showing a combination of motion detection information from the image processing chips 15X, 15Y, and 15Z regarding translation and rotation of the back of the hand.

FIG. 4 is a diagram illustrating another configuration example of an operation input device in which an image processing chip on which an image space sensor is mounted is arranged.

FIG. 5 is a flowchart for explaining a motion detection operation of the operation input device according to the embodiment.

FIG. 6 is a flowchart illustrating details of a motion determination process in step S6 of FIG. 5;

[Explanation of symbols]

 7I1, 7I2, 7I3, 7I4 One-axis angular velocity sensor 7M1, 7M2, 7M3, 7M4 One-axis angular velocity sensor 7TY, 7TX One-axis angular velocity sensor 12X, 12Y Two-axis acceleration sensor 12Z One-axis acceleration sensor 14X, 14Y, 14Z Angular velocity sensor 15X, 15Y, 15Z Image processing chip

Claims (3)

[Claims] 1. A motion detecting apparatus for detecting a motion of an object to be mounted using at least one of an acceleration sensor and an angular velocity sensor, wherein an image capturing means for capturing an image of a surrounding is compared with images captured at different times by the image capturing means. A motion detecting device, further comprising: a comparing unit that performs the operation, and an obtaining unit that obtains motion information of the mounted object based on a comparison result of the comparing unit. 2. An operation which is mounted on the back of an operator's hand and detects at least one of an acceleration and an angular velocity applied to the hand, and an operation of generating a command in a predetermined method using a detection result of the back detection means. An operation input device having an input analysis unit, an imaging unit that captures an image of a surrounding, a comparison unit that compares images captured at different times by the imaging unit, and a hand movement based on a comparison result of the comparison unit. An operation input device, further comprising: acquisition means for acquiring information. 3. A backrest detecting means mounted on the back of the operator's hand to detect the movement or posture of the back of the operator's hand, and a finger mounted on the operator's finger and detecting the posture of the operator's finger A posture detecting means, a hand shape estimating means for obtaining a shape of the entire hand of the operator based on outputs of the back of hand detecting means and the finger posture detecting means, and a predetermined shape using an output of the hand shape estimating means. An operation input device including an operation input analysis unit that generates a command, an imaging unit that captures an image of a surrounding, a comparison unit that compares images captured at different times by the imaging unit, and a comparison result of the comparison unit. And an acquisition means for acquiring hand movement information.