JP2832333B2 - Object shape / posture detection device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus for detecting the shape and orientation of an object, and more particularly to an apparatus for detecting the shape and orientation of an object in a three-dimensional space based on image information in two directions.

[0002]

2. Description of the Related Art Research and development of a virtual space presentation system has been promoted as a system in which people far away from each other can hold a communication conference with a sense of reality.

[0003] The virtual space presentation system is a system for presenting a virtual space virtually formed by a computer system as a place for a communication conference.

[0004] However, the virtual space presentation systems proposed so far often require the user of the system to wear a special device such as an HMD (head mounted display) for displaying stereoscopic images. There was a disadvantage that there is.

[0005] In addition, in order to realize a hand motion in a virtual space, the actual hand motion of a user is detected. When recognizing a hand gesture, it is common for a user to wear a special glove to which a sensor is added. Therefore, in that case, there is a problem that the burden on the user is large because such gloves are complicated to put on and take off.

[0006] On the other hand, a vision-based hand gesture recognition device capable of recognizing a hand gesture similar to that when wearing such special gloves can be realized without wearing such special gloves. Is being promoted. A vision-based hand gesture recognition device is a device that recognizes hand gestures using hand image information obtained by an imaging device such as a camera.

Nakajima et al. Have proposed a method of tracking the movement of a finger with a single camera (hereinafter referred to as a single eye) using a model consisting of a fingertip and the base of the finger (" Finger Motion Detection Method "Graphics and CAD6
7-6, pp. 41-46, 1994. ). In that method, a special marker is attached to a finger in order to detect a feature point necessary for detecting the movement of the finger, and the finger is imaged with a single eye.

However, this method has a drawback that it is troublesome to attach a special marker.
Since imaging is performed with a single eye, there is a disadvantage that the restoration accuracy is not high. Here, the restoration accuracy refers to the accuracy when an object in a real space is restored in a virtual space.

Have proposed a method for performing three-dimensional reconstruction with a single eye using a time-series binary image (see "Virtua").
l Reconstruction of fist movement for reality "IEICE Technical Report PRU93
-57, pp. 23-30, 1993. ). In that method, posture restoration by associating multiple frames is possible. However, this method has a disadvantage that a cumulative error increases.

Also, Okamura et al. Prepare in advance a feature amount such as a finger length corresponding to each of several types of gestures, and collate the feature amount with the detected feature amount. We proposed a method to recognize gestures by doing ("Non-contact hand shape recognition and its application" IEICE HC93-
6, pp. 31-38, 1993. ). However, in such a method, the main focus is on performing gesture recognition, and there is a problem that the posture of the hand cannot be detected continuously.

Therefore, a system capable of solving various difficulties as described above has been proposed (Ishibuchi et al .: "Real-time hand shape recognition using image processing and application to man-machine interface"). Proceedings of IEICE Fall Meeting, pp. 1-132-132, 1991. and Ishibuchi et al .: "Real-time Hand Shape Recognition Using a Pipelined Image Processing Device" IEICE Spring Conference Proceedings. ,
pp. 1-21-1-1-291, 1992. ).

In this system, the palm is regarded as a planar shape, and the positions and positions of the center of gravity and the fingertips of the hand, which are characteristic points, are represented in the plane, thereby expressing the shape and posture of the hand. Model is used. Then, these feature points are tracked by image information obtained from the two cameras, and processing including motion restoration is performed in real time.

In the system, it becomes possible to perform the processing including the need for attaching the marker and the motion recovery in real time. Further, in the system, by performing a fingertip matching process between two images captured by two cameras, it becomes possible to cope with a hand rotation operation.

[0014]

However, in the method of performing such processing including motion restoration in real time, the camera axis (the central axis of the camera image) is controlled by the hand movement.
Under such conditions that the plane of the palm is parallel, there is a possibility that the fingertip point cannot be obtained correctly. Therefore, in that case, there is a problem that the shape and posture of the hand cannot be detected stably.

The present invention has been made in order to solve such a problem, and provides an object shape and orientation detection apparatus capable of stably detecting the shape and orientation of an object such as a hand. The purpose is to do.

[0016]

According to a first aspect of the present invention, there is provided an object shape / posture detecting apparatus for detecting a shape and a posture of a movable object which can be regarded as a planar shape. Image pickup means and image information selection means.

The detecting means detects the shape and orientation of the object in a three-dimensional space based on image information in two directions obtained by imaging the object from two directions.

The at least three image pickup means are arranged toward the object from different directions, each of which picks up an image of the object around an image pickup axis toward the object and outputs image information obtained thereby.

[0019] The image information selecting means may select an image having a small angle formed by a normal vector of the plane of the object obtained from the information on the posture detected by the detecting means and a predetermined vector of the imaging axis of each imaging means. The two image pickup means are selected in order, and the image information in two directions output from the selected two image pickup means is provided to the detection means for detecting the shape and posture of the object.

According to a second aspect of the present invention, in the first aspect, the image information selecting means includes an inner product calculating means and an image selecting means.

The inner product calculating means obtains the inner product of the normal vector of the plane of the object and the vector of the imaging axis of each imaging means.

The image selection means selects two imaging means in descending order of inner product based on the calculation result of the inner product calculation means, and detects two-directional image information output from the selected two imaging means. Give to.

[0023]

According to the first aspect of the present invention, an object is imaged from different directions by at least three imaging means. Based on the image information in two directions among the image information obtained by the imaging, the detecting unit detects the shape and posture of the object in the three-dimensional space.

In this case, the image information is selected by the image information selection means by using the normal vector of the plane of the object obtained from the information of the posture detected by the detection means and the predetermined imaging axis of each imaging means. This is based on the angle between the vector and the vector. That is, when such an angle is small, the imaging means can appropriately image the object. Therefore, the image information selection means selects two imaging means in ascending order of the angle, and outputs two directions. Is given to the detecting means.

Therefore, when the object moves, the image information given to the detecting means for detecting the shape and orientation of the object is determined by the image information of the two image capturing means located at a position suitable for capturing the moving object. Always switched to information.

According to the second aspect of the present invention, the normal vector of the plane of the object obtained from the posture detected by the detection means by the inner product calculation means and the predetermined imaging axis of each imaging means. The inner product with the vector is determined. The imaging means having a large inner product includes a normal vector of the plane of the object,
Since the angle formed by the vector of the imaging axis of the imaging means is small, the object can be properly imaged.

For this reason, the image information in the two directions of the two imaging units having a large inner product selected by the image selection unit is given to the detection unit as image information captured from a position suitable for imaging the object.

[0028]

Next, an embodiment of the present invention will be described in detail with reference to the drawings.

First, an outline of a real-time shape / posture detecting apparatus according to an embodiment of the present invention will be described. This device is
The palm is regarded as a planar shape, and the positions of the center of gravity of the hand and the fingertip points, which are characteristic points, are represented in the plane to represent the shape and posture of the hand, and the image information obtained by a plurality of cameras is used. Based on this, the plane and the movement of the feature points in the plane are tracked.

A feature of the apparatus according to this embodiment is that two optimal cameras are selectively used in response to movement of a hand whose shape and posture are to be detected. That is, the camera that obtains the image information is switched according to the hand movement.

The image information in two directions obtained from the two cameras thus selected is used for detecting the shape and posture of the hand. The detection result of the shape and posture of the hand obtained in this embodiment is used as data of the virtual space presentation system.

FIG. 1 is a conceptual diagram showing the arrangement of cameras in a real-time shape / posture detecting apparatus according to an embodiment of the present invention.

Referring to FIG. 1, a palm, which is an object including a planar shape, exists on a horizontal plane H as a palm plane 10. Three or more cameras 1, 2, 3, ..., k are fixedly arranged around the palm plane 10. Each camera is arranged with the camera axis (imaging axis) facing the palm plane 10. Cameras 1, 2, 3, arranged in this way
, K obtain an image including the palm plane 10 and its background by imaging a predetermined range centered on the camera axis.

As described above, a large number of cameras 1, 2, 3,
, K, the palm plane 10 is imaged from many directions. Image information of the palm plane 10 obtained by such imaging is transmitted from each camera to a detection unit (described later).

Next, the overall configuration of the shape / posture detecting apparatus according to this embodiment will be described. FIG. 2 is a block diagram showing the configuration of the shape / posture detecting device according to the embodiment of the present invention.

FIG. 2 shows a configuration in which three cameras shown in FIG. 1 are provided. Referring to FIG. 2, this device includes three cameras 1, 2, 3, an image information selection unit 4 and a detection unit 5.

The camera 1 includes an imaging section 11 and an area dividing section 1
2. It includes a spindle detection unit 13, a rotation conversion unit 14, and a fingertip candidate point detection unit 15. Each of the cameras 2 and 3 has the same configuration as the camera 1. Therefore, the description is omitted here.

The image information selecting section 4 includes an inner product calculating section 41 and an image switching section 42. The detection unit 5 includes a center of gravity / main axis three-dimensional restoration unit 51, a fingertip point three-dimensional restoration unit 52, and a posture estimation unit 5.
3 and a fingertip position estimating unit 54.

First, an outline of the operation of the apparatus will be described.
In each of the cameras 1 to 3, based on image information obtained by imaging a hand (palm plane 10) to be imaged, image information that has been rotationally transformed so that the fingertip of the hand faces upward on the image. , And image information indicating the detection result of the fingertip candidate point used for estimating the position of the fingertip later.
The image information thus obtained is provided to the image information selection unit 4 from each of the cameras 1 to 3.

The image information selecting section 4 selects image information from two cameras (image information in two directions) from the image information provided from the cameras 1 to 3, and detects the selected image information. Give to 5. The selection state of the image information in the image information selection unit 4 is switched based on information related to the posture of the palm plane 10 given from the detection unit 5.

The detecting unit 5 detects the posture of the palm by estimating the posture of the palm plane 10 based on the image information in two directions given from the image information selecting unit 4, and detects the position of the fingertip of the hand. Is detected to detect the position of the fingertip.

Next, the detailed operation of each part of the apparatus will be described. The cameras 1 to 3 operate as follows.
Since each of the cameras 1 to 3 has the same configuration and performs the same operation, the operation of the camera 1 will be described here as a representative example.

In the camera 1, the imaging section 11 images the hand (palm plane 10) to be imaged. Image information obtained as a result of the imaging is provided to the area dividing unit 12. The region dividing unit 12 divides an image obtained from the given image information into two regions, a hand region and a background region. The division of the area is performed using the color information and the luminance information included in the image information as determination criteria. The image information segmented by the segmentation unit 12 is provided to the spindle detection unit 13 and the fingertip candidate point detection unit 15.

The main axis detecting section 13 obtains an average edge direction which is an average direction of the edge of the fingertip based on the given image information. The average edge direction is recognized as a hand direction. Here, the main axis refers to the direction in which the middle finger of the hand faces. The image information indicating the result of the processing in the spindle detection unit 13 is provided to the rotation conversion unit 14.

The rotation converter 14 performs a process of rotationally converting the image of the hand so that the fingertip of the hand is directed upward on the image, based on the information on the direction of the hand given from the main axis detector 13.

In the fingertip candidate point detecting section 15, the area dividing section 1
1 and the three-dimensionally restored image information after rotation conversion given from the detection unit 5 (centroid / main axis three-dimensional restoration unit 51 described later) based on the image information given by ) Are extracted. The extraction is
This is performed by a labeling process.

The labeling process is a process of searching for a hand region in order from the top of the image on the rotationally transformed image, and regarding the upwardly convex portion as the tip of the fingertip. Information on the fingertip candidate points from which such extraction has been performed is output from the fingertip candidate point detection unit 15 as a detection result of the fingertip candidate points.

In the camera 1 in which such processing is performed, the image information output from each of the rotation conversion unit 14 and the fingertip candidate point detection unit 15 is both provided to the image information selection unit 4.

The image information selecting section 4 selects two cameras from which the optimum image information can be obtained from the image information given from the cameras 1 to 3, and selects the image information from the selected two cameras. Are given to the detection unit 5 respectively. The detailed operation of the image information selection unit 4 will be described after the operation of the detection unit 5 is described.

The detecting section 5 performs the following operation.
The detection unit 5 processes image information provided from the two cameras in parallel.

The center-of-gravity / main axis three-dimensional restoration section 51 receives image information given from the rotation conversion sections of each of the two selected cameras. The centroid / spindle three-dimensional restoration unit 51 three-dimensionally restores the rotationally transformed image. Where 3
Dimensional restoration means that a two-dimensional image obtained by imaging is
It refers to the process of converting (restoring) to a two-dimensional image.

The center-of-gravity / main-axis three-dimensional restoring unit 51 performs a process of three-dimensionally restoring the direction of the main axis of the hand and the center of gravity, which are rotationally converted, based on the image in two directions. The three-dimensionally restored image information is output to the fingertip candidate point detection unit (15,
25, 35) and a posture estimating unit 53.

The fingertip point three-dimensional restoring unit 52
Given from the fingertip candidate point detection units of each of the two cameras
A process of three-dimensionally restoring the image information in the direction is performed.

The three-dimensional restoration is performed by matching feature points between images in two directions.
The matching is based on Koichi Ishibuchi's “Real time
handgesture recognition using 3d prediction mode
l "In International Conference on Systems, Man and
This is performed using the confidence table shown in Cybernetics, pp.324-328, 1993.

The certainty table is a table of certainty functions indicating the degree of correspondence in matching when feature points are matched between images in two directions.

There are the following types of such confidence tables, for example. That is, there are a certainty table relating to the distance from the center of gravity to a certain feature point, a certainty table relating to the arrangement of the fingers, a certainty table indicating the accuracy when matching is performed, and the like.

The posture estimating unit 53 receives the three-dimensionally restored image information from the three-dimensional restoring units 51 and 52, respectively. In the posture estimating unit 53, based on the given information,
A process of estimating the posture of the palm plane 10 (palm) is performed using information on the position of the center of gravity in three dimensions, the principal axis vector (the direction of the principal axis in the three-dimensional space), and the rotation angle of the palm plane 10 around the principal axis.

The fingertip position estimating unit 54 receives the image information from the fingertip candidate point detecting units and the image information from the posture estimating unit 53 of each of the two selected cameras. The fingertip position estimating unit 54 performs a process of projecting fingertip candidate points onto the palm plane 10 whose posture has been estimated by the posture estimating unit 53 and estimating the fingertip position.

As described above, the detecting unit 5 can detect the position and the posture of the hand based on the image information in two directions selectively given from the image information selecting unit 4.

Next, the operation of the image information selecting section 4 will be described. In the image information selection unit 4, the image switching unit 42
It receives image information given from each of the cameras 1 to 3. Further, the inner product calculation unit 41 receives information on the posture of the palm plane 10 estimated by the posture estimation unit 53.

The vector c1 #, c2 #, c3 # of each camera axis of the cameras 1 to 3 is stored in the inner product calculation section 41 in advance. Here, ↑ indicates a vector symbol. This is the same in the following description. In the inner product calculation unit 41, the vectors c1 ↑, c
2} and c3}, and the respective inner products of the normal vector p of the palm plane 10 obtained based on the posture information of the palm plane 10 estimated by the posture estimating unit 53 (vector p1 ↑ · vector c1 ↑, vector p ↑ · vector c
2 ↑, vector p ↑ · vector c3 ↑). The calculation result of such inner product is provided from the inner product calculation unit 41 to the image switching unit 42.

The image switching unit 42 selects two cameras in descending order of the value of the inner product based on the calculation result of the inner product given from the inner product calculation unit 41, and selects the selected two cameras.
Image information corresponding to each of the cameras is provided to the detection unit 5. The condition for selecting a camera in accordance with the calculation result of the inner product is given by the following equation (1). The following expression (1) is a conditional expression when n cameras are provided.

P ↑ · ci ↑ + p ↑ · cj ↑ → Max (1) In the formula (1), when a combination of adding two inner products is considered, the combination of the two inner products becomes the largest (Max). (2) indicates that two cameras corresponding to the combination are selected.

In other words, in the image information selecting section, the vectors c1ｃ and c2 ↑ of the camera axes of the cameras 1, 2, and 3
And c3} and the normal vector p ↑ of the palm plane 10
The two cameras are selected in order from the smaller angle among the angles made by the two cameras, and the image information corresponding to those cameras is provided to the detection unit 5.

In the shape / posture detecting device in which such an operation is performed, even if the palm plane 10 moves due to the movement of the hand to be imaged, an image is taken from an optimum angle according to the posture of the palm plane 10. Cameras are selected. For this reason, this shape / posture detection device can stably detect the shape and posture of the object to be imaged even if the posture of the palm plane 10 changes.

Next, the results of experiments conducted to clarify the effects of the present invention will be described. First, the conditions of the experiment will be described.

In the experiment, three cameras were used. These cameras were radially arranged on a vertical plane including a portion where a hand to be imaged was located. Each of those cameras was fixed towards the hand. The background portion of the hand is set to black so that the hand region can be extracted only with the luminance.

Then, the hand was opened, and the camera was rotated around the normal to the plane where the cameras are arranged in the vicinity of the center where the three cameras faced. Experiments were conducted in two ways: an imaging method using a conventional device that captures images with only two cameras, and an imaging method using the device of the present invention that captures images by automatically switching between three cameras.
The rotation angle of the palm was detected in each of the following methods.

FIG. 3 shows the experimental results. FIG. 3 is a graph showing experimental results of an experiment in which the rotation angle of the hand is detected by comparing the apparatus according to the present invention with the apparatus according to the related art. In FIG. 3, as the data obtained by the experiment, the rotation angle D2 detected when only two cameras are used (in the case of the conventional example) and when three cameras are switched and used (this application). The relationship between the rotation angle D3 detected in the case of the invention), the number of the selected camera No. 1 to No. 3 when three cameras are switched and used, and the process steps (0 to 160) are shown. It is. In the selected camera, the selected period is indicated by a bold solid line in a portion corresponding to the camera number.

As is apparent from FIG. 3, when only two cameras are used, the detection angle becomes constant regardless of the rotation of the hand during the middle of the process steps. This is because the detection accuracy has been reduced by occlusion.

On the other hand, when three cameras are switched and used, the cameras are switched so that the rotation angle can be accurately determined even in the middle of a process step in which it is difficult to detect with only two cameras. Was detected.
Thus, according to this experiment, the shape and
It turned out that the attitude detection device is superior to the conventional one.

[0072]

According to the first aspect of the present invention, the image information in two directions given to the detecting means for detecting the shape and posture of the object is a position suitable for imaging the moving object. Is always switched to the bidirectional image information from the two image pickup means. Therefore, it is possible to stably detect the shape and the posture of the object in the three-dimensional space.

According to the second aspect of the present invention, the normal vector of the plane of the object obtained from the information on the posture of the object detected by the detecting means, and the vector of the predetermined imaging axis of each imaging means From the two imaging means with a large inner product of
The image information in the direction can be provided to the detection unit as image information captured from a position suitable for capturing an object.

[Brief description of the drawings]

FIG. 1 is a conceptual diagram showing an arrangement state of cameras in a real-time shape / posture detecting apparatus according to an embodiment of the present invention.

FIG. 2 is a block diagram showing a configuration of a real-time shape / posture detection device according to an embodiment of the present invention.

FIG. 3 is a graph showing an experimental result of an experiment for detecting a rotation angle of a hand in a device according to the present invention and a device according to a conventional example.

[Explanation of symbols]

 1, 2, 3 camera 4 image information selection unit 5 detection unit 10 palm plane 41 inner product calculation unit 42 image switching unit

Claims (2)

(57) [Claims] An object shape / posture detection device for detecting the shape and orientation of a movable object that can be regarded as a planar shape, based on image information in two directions obtained by imaging the object from two directions. Detecting means for detecting the shape and orientation of the object in a three-dimensional space; and detecting means for arranging the object from different directions toward the object, each of which images the object about an imaging axis toward the object. At least three image pickup means for outputting image information obtained thereby, a normal vector of a plane of the object obtained from information of the posture detected by the detection means, and a predetermined normal The two imaging units are selected in order from the one with the smallest angle between the vector of the imaging axis and the selected 2
An object shape / posture detecting apparatus, comprising: image information selecting means for providing image information in two directions output from two image pickup means to the detecting means for detecting the shape and posture of the object. 2. The image information selecting means includes: an inner product calculating means for obtaining an inner product of a normal vector of a plane of the object and a vector of an imaging axis of each of the imaging means; 2. An image selection unit for selecting two imaging units in descending order of the inner product on the basis of the inner product, and providing two-direction image information output from the selected two imaging units to the detection unit. An object shape / posture detection device according to the description.