JP6810442B2 - A camera assembly, a finger shape detection system using the camera assembly, a finger shape detection method using the camera assembly, a program for implementing the detection method, and a storage medium for the program. - Google Patents

Description

The present invention relates to a method of detecting the shape of a human finger from an image captured by a camera, and in particular, estimates the shape of the finger that is not reflected from a plane (2D) image of a single lens (monocular) camera. Regarding the system and method of detecting.

Conventionally, as an example of a method for driving a multi-finger type robot hand or a manipulator, or for operating the fingers of a character such as a game or animation displayed on the display unit, from the movement of the user's hand. Gesture input for detecting the shape of the finger or "motion capture of fine finger movement" (motion capture of finger movement = hand mocap: three-dimensional finger shape estimation technology) is known. For hand mocap, there is a request to acquire data in association with the motion capture of the whole body together with the facial expression, but the facial expression is drawn as a computer animation in real time in combination with the motion capture of the human body movement. Since the system has become practical, it is said that only hand mocap is the last technology that has not yet reached the level of practical use.

The hand morphs can be roughly classified as follows: (X) A device mounting method that obtains the shape of a finger from the output of a sensor device mounted on the user's arm or finger, or the analysis result of image data obtained by capturing a marker with a camera. , (Y) It can be classified into two types of image processing methods that detect the hand morphing motion of the finger shape from the captured images of the arm and fingers. (X) In the case of the device-mounted type, accurate finger detection is possible, but the configuration is large, preparation is time-consuming and detection is not easy, and in the case of sensor-mounted, the user is restrained and freely Sometimes it couldn't work. Therefore, the (Y) image processing method is desirable for easily detecting the fingers. Hereinafter, the (Y) image processing method related to the present invention and the technique related thereto will be mainly described.

The (Y) image processing method is roughly classified into (Y1) 3D-model-based approach (hereinafter referred to as 3D approach) and (Y2) 2D-appearance-based approach (hereinafter referred to as 2D approach). ) Can be classified into two methods.

The (Y1) 3D approach requires a plurality of lens mechanisms capable of simultaneously photographing the fingers from a plurality of different directions, and a computer having a high computing power for estimating a smooth moving image of the finger shape in real time. In the (Y2) 2D approach, for example, the contour line (outline of the silhouette) information of the hand image is featured and matched by the (Y2a) higher - order local autocorrelation feature (hereinafter referred to as HLAC). It is possible to estimate with high accuracy, but if the contour lines are the same or similar, it is difficult to identify the case of different shapes, and even if the shape of the same hand is different, the contour lines are different and other hands. The problem of individual differences, which makes it difficult to distinguish from the shape, cannot be solved. (Y2b) In the method based on Histogram of Gradients (hereinafter referred to as HoG), since the luminance gradient information of the image is featured, the internal range of the contour line shape can be identified and individual differences can be solved. A large amount of memory is required, and individual differences cannot be handled at the feature quantification level. Hereinafter, the (Y2) 2D approach related to the present invention, the image processing method by (Y2a) HLAC, and the technology related thereto will be mainly described.

As the camera of the (Y2) 2D approach, a monochrome or color monocular camera (camera) is used, but it is difficult to detect the finger shape from the monocular (2D) captured image.

Further, in the (Y2) 2D approach, for example, the joint angle and rotation angle data of the fingers obtained by the (X) device mounting method, and the image features from the contour lines of each divided region of the grayscale finger image captured by the monocular camera. It is known that an image database for collation is created by combining with a quantity, and the collation result of a new finger image for finger shape detection and a finger image of the database is used as a finger detection result (for example). See Patent Document 3). In that case, among the hand images in the image database for the image feature amount such as the contour line obtained from the new hand image, the image data having the most similar image feature amount is searched, and the finger combined with the most similar image data is searched. The accuracy of finger shape detection is improved by estimating the finger shape of a new image from the joint angle and rotation angle data of the above.

Further, in the (Y2) 2D approach, for example, in order to reduce the amount of image data in the image database for collation and facilitate collation, for example, the extension direction of the forearm from the contour line of the forearm of each finger image and the wrist. The orientation of the finger images in the image database can be aligned by finding the position of and collating with the wrist to the tip in the same orientation. Further, the size can be made uniform by normalizing each finger image having a predetermined number of pixels in the vertical and horizontal directions by using the contour line of each finger image (see, for example, Patent Documents 1 and 2). Therefore, in the conventional method of estimating the shape of a finger using a monocular camera, the contour line information is obtained as precisely as possible from the raw data of the hand image, and the shape of the finger is estimated from the contour line information and the image data of the image database for collation. Was there.

When detecting the "finger shape, positional relationship, and movement" by the conventional (Y2a) HLAC from the image captured by the monocular camera using only the contour line shape, the following (a), (b), and (c) It is known that it is difficult to accurately estimate the finger shape because it is difficult to distinguish when the finger shape is different but the contour line is the same or similar for the above three reasons. Further, due to individual differences, even if the same finger shape is used, the contour line differs for each person, and there is a problem that it becomes difficult to distinguish one person's finger shape from another person's finger shape.
(A) Since the fingers have an articulated structure, the shape change is complicated.
(B) When a joint is bent or gripped, the contour line shape of the finger is that the finger is often hidden behind the back of the finger or the palm of the finger.
(C) The finger has a large movable space, although the ratio of the part to the whole body is small.

In addition, when creating an image database by associating each image of the (Y2) 2D approach using the monocular camera with joint angle data, for example, the thickness and length of each finger and how the thumb comes out from the back of the hand. Since there are various individual differences in the size of the back of the hand, it is difficult to create a finger image that greatly covers the individual differences, such as a general-purpose or representative sample finger image or a reference finger image. Therefore, it is necessary to prepare a large number of image data in consideration of individual differences, and the amount of image data stored in the image database of finger images has increased. Then, in order to reduce the amount of image data stored in the image database, image data including individual differences within a certain range was prepared and estimated, but if the individual differences are larger than the assumed range, erroneous estimation is performed. I had something to do.

In addition, preparing all the possible image data in response to various individual differences in order to avoid the above-mentioned misestimation greatly increases the amount of data, and as a result, the amount of memory required. Will also increase, and the man-hours for creating a database will also increase. Further, the amount of data to be collated with new image data increases, it takes time to search the image database for the most similar image data, and the calculation speed of the data processing device may be insufficient. Alternatively, conversely, the amount of data for collation processing may be limited due to the limitation of the processing capacity of the data processing device, and it may be difficult to prepare all the image data.

Further, as described in Patent Document 2, the amount of image data is reduced and the calculation speed is insufficient by using the ridge line shape passing through the center of the finger instead of the contour line used for the image feature amount. It is also known that can be eliminated. As the ridge line of the finger, for example, a grayscale image of the finger captured by a monocular camera of the user is subjected to a pseudo skeletonization process by using a thinning process used in edge processing or the like to form the skeleton. Use the thin line (ridge line). In addition, the tip of noise other than the fingertip at the time of thinning is excluded as invalid if the distance from the center of gravity coordinates of the finger is within the same value. Further, in the detection of the movement direction and the movement amount of the finger in the above-mentioned 2D approach method, the movement direction of the finger is detected by using the three-dimensional shape estimation of the finger (hand pose establishment) or the like from the contour line shape of the finger image described above. And the amount of movement (hand tracking) may be detected.

Further, as an aid for solving the problems of Patent Documents 1 and 2, data including a plurality of finger shapes and image feature amount data including joint angles and rotation angles for collation by a device-mounted hand mocap. A database for collation of the set is created in advance, and the captured image is normalized and smoothed by the information processing device from the captured image for detection to generate the image feature amount data for detection, and the collation image feature amount in the data set. It is known that a data set containing similar image feature data for collation is selected by comparison with the data, and the finger shape data in the selected data set is included in the finger shape detection result and output. (See, for example, Patent Document 3).

Also, as an aid to solving the problems of Patent Documents 1 and 2, first, one hand (for example, the right hand) is photographed in advance with a camera assembly installed on the back side, and the data is stored in the other hand (for example, the left hand). Wear the glove, perform the same operation with the right and left hands, and store both the finger image data from the back side camera assembly and the finger angle data of the data glove in the database. At the time of detection, one hand (for example, the right hand) is photographed by the back side camera assembly, the hand finger angle data in the database is read from the hand image data of the back side camera assembly, and the hand robot or the CG image of the hand is operated. It is known to output finger angle data to (see, for example, Patent Document 4).

Further, as the camera assembly of the back part, (a) a flat support base part is formed on the surface on the back side of the back part, and (b) it is made of a material having flexibility that can correct the angle by hand. A support column having a camera mounting portion at the tip is provided so as to project from the support base portion toward the sky, and (c) both the support foundation portion and the back of the hand cross from the little finger side to the index finger side of the back of the hand. It has a flexible fixing band that wraps around the back of the hand so that (d) the camera is installed at the tip of the support column, and the camera's imaging lens can image the fingers diagonally from the front between the thumb and index finger. As described above, it is known that the camera corrects the angle of the camera support column (see, for example, Patent Document 4).

International Publication WO2009 / 147904 Pamphlet International Publication WO2013 / 051681 Pamphlet JP 2016-014954 JP 2015-100697

However, the conventional shape detection methods using a monocular camera fixed indoors or fixed to a rack on a desk in Patent Documents 1 to 3 have the following problems.
(A) The angles of the direction of the camera lens with respect to the fingers of the subject are very diverse and the angle range is widened, and as a result, the poses of the captured images are also diverse, and the amount of data in the database is increasing.
(B) In addition, the distance between the subject's fingers and the camera lens is also very diverse, and if it does not fit within the depth of field, it is necessary to adjust the focal length and angle of view, resulting in the scale of the captured image. The ratio was also diverse, and it was necessary to adjust the enlargement / reduction of the image, and the amount of data in the database was increasing.
(C) Furthermore, although it is related to the variety of poses of the captured image of (A) above, there are also captured images in which only the back of the hand does not show the base of the fingers and the back of the hand is not shown. It is difficult to estimate the fingers, and the improvement of collation accuracy and the improvement of individual differences are also partial. In each shape estimation using the collation results, the finger joints in the finger image have the same contour in the flexed state. Improvements in identifying irregular shapes with lines were also partial.
(D) Further, collation using the contour line, the ridge line, and the normalized and smoothed captured image from the conventional captured image requires a lot of time due to the large amount of data in the database as described above. ..

In addition, for example, in performing arts, various dances, or sports where artistic points are evaluated (ice skating, synchronized swimming, rhythmic gymnastics, etc.), in addition to the movement of the entire human body, the movement of the hands as well as the facial expression is also evaluated. Included in. Also, in games and animations, in addition to the movement of the entire human body, the movement of the hand as well as the facial expression is an important drawing element. However, in the conventional fixed cameras of Patent Documents 1 to 3, even if the number of cameras is increased to a plurality of cameras, the fingers may not be imaged or the entire arm and the tip may not be imaged. It was not possible to acquire data at the same time as motion capture of the entire body.

In the conventional camera assembly of Patent Document 4, although the contents are not described in detail in the specification, the possibility of improving the above-mentioned problems (A) to (D) and the hand mocap of the whole body It has the potential to be able to acquire data at the same time as motion capture. However, in the shape detection method using the camera assembly, the data glove is used for inputting the finger angle data and the like stored in the collation database as in Patent Documents 1 to 3, and the finger angle data corresponding to the finger image is used. It took a lot of manpower and time to create a data set that matched the above. In addition, patent documents 1 to 3 indicate that the amount of data in the database has increased because there are various individual differences in human hands such as finger thickness, thickness, and length when detailed feature quantification is performed. It was similar.

Therefore, the present invention provides a device that can estimate all the joint angles of the fingers for each imaging frame by inputting the captured image data from one small camera attached to the back side of the hand, and further, the number of poses of the finger images in the collation database. It has the effect of reducing the reduction / enlargement ratio, reducing the amount of data, and acquiring data at the same time as motion capture of the entire body with hand morph, and further streamlining the creation of the collation database for creation. It is an object of the present invention to provide a finger shape detection method and system that can reduce time and labor and further suppress individual differences.

First, in order to solve the above-mentioned problems, as described above, in the camera assembly of the present invention, one or more small cameras and a support portion capable of installing the camera in the sky near the back of one hand of the wearer are provided. (A) The camera has a rod shape so that substantially the entire one hand of the wearer is accommodated within the angle of view of the camera and has a length equal to or longer than the focal distance at that time. A support column formed of a material having at least the characteristics of both self-supporting strength even when stress due to the weight of the camera is applied and flexibility that can change the angle, and (b) the support column on the surface of the back of the camera. It can be connected and installed so that the support column can be supported by protruding from the top, and is formed of a hard material in a plane along the surface of the back of the hand and having an area equal to or larger than a predetermined area. A support base portion having a flat surface portion, (c) a camera mounting portion provided at the tip end portion of the support column and capable of mounting the camera, and (d) the support base portion arranged on the surface of the back portion. A band shape wound around the back of the hand in a direction crossing the base of each finger of the back of the hand or an upper surface of the support base so that it can be pressed and fixed from above at least a part of the back of the hand. With a material that has at least the characteristics of both the strength that does not break during pressing and fixing and the flexibility that allows it to bend along the surface of the back of the hand so that the shape of the back of the glove can be pressed and fixed. It has a back fixing portion to be formed, and the support column is attached to the back portion of one hand of the wearer, and then is on the fingertip side of a line crossing the base of each finger in the back portion. And, by utilizing the flexibility of the material, at least the angle and the angle of protrusion from the support base portion so that the camera can image substantially the entire image of the one hand from between the thumb and the index finger of the wearer's one hand. A camera assembly in which the angle at which the camera is directed in the imaging direction is changed.

Preferably, in the camera assembly according to the present invention, the camera mounting portion is the mounting position of the camera, and one hand of the wearer is within the focal length including the depth of field of the camera. It is possible to slide back and forth along the optical axis of the camera so that the total vertical dimensions of the fingers and back of the frame roughly match the vertical dimensions of the image frame, and the mounting angle of the camera is captured. It is possible to adjust the rotation of the captured image around the optical axis of the camera so that the tilt angle of the image frame is a predetermined angle that matches the image frame to be collated stored in advance. You may do so.

Preferably, in the camera assembly according to the present invention, the flat surface portion of the support base portion faces the left, right, front and rear ends of the back surface portion along the surface of the back surface portion centering on the connection portion with the support pillar. It may be arranged so as to be stretched.

Preferably, the finger shape detection system according to the present invention is a system having the above-mentioned camera assembly and at least one information processing device to detect the finger shape, and the information processing device is The image storage unit that stores the image data of each finger shape input from the camera and the computer graphics (CG) program have different dimensional ratios for each part of the fingers and the dimensional ratio of the back of the hand, and the pose of the fingers. Of the various hand image forming parts that form the hand image of various CG images that are different from each other and the various hand image formed by the virtual hand image forming part, the dimensional ratio of each part of the hand and the dimensional ratio of the back of the hand are used as a reference. , A plurality of virtual finger images having the same dimensional ratio but different finger poses are classified into the same group, and the dimensional ratio data and the angle data for each finger joint in the pose are obtained as virtual. The virtual finger image database stored as a data set containing at least each finger image, the dimensional ratio of each part of the finger and the dimensional ratio of the back of the hand of the captured finger image are discriminated, and classified into the virtual finger image database for each group. From the plurality of virtual finger images stored as a data set together with the respective dimension ratio data, a group discriminating unit that determines which of the plurality of virtual finger images in the group is read out from the respective dimension ratio data. And a silhouette image forming unit that forms a silhouette image having contour lines for both images of the imaged hand image stored in the imaged hand image storage unit and the virtual hand image stored in the virtual hand image database. , The ridge line image forming portion that forms a ridge line image from any of the captured finger image, the virtual finger image, or the silhouette image, and the silhouette image and the ridge line image are superimposed to form a superimposed image. Whether all the ridge lines of the ridge line image are accommodated in the internal range of the superimposed image forming portion and the silhouette image portion of the superimposed image, or is there the ridge line protruding from the external range of the silhouette image portion? When there is an out-of-range ridge line determination unit that determines the data and the ridge line that extends beyond the external range of the silhouette image portion, the virtual finger image in the original image of the image included in the superimposed image is estimated in the latter stage. Input imaging hand fingers for each virtual hand image of the candidate image narrowing unit to be excluded from the candidate images used for processing and the candidate image narrowed down by the candidate image narrowing unit. The similarity with the image is calculated, the virtual finger image having the minimum similarity is determined, and the angle data for at least each finger joint is included from the data set including the virtual finger image having the minimum similarity. By selecting data, it has at least a finger angle estimation unit that performs estimation processing including the similarity calculation, and a data output unit that outputs the result of the estimation processing to a device to be controlled.

Preferably, in the finger shape detection system according to the present invention, the total upper and lower parts of the fingers and the back of the hand in the imaged image frame are raised and lowered with respect to the image data of each hand shape stored in the imaged hand image storage unit. The image dimensional adjustment unit that enlarges or reduces the dimensions so that the dimensions substantially match the vertical dimensions of the image frame and the tilt angle of the imaged image frame match the image frame to be collated stored in the storage device in advance. An image angle adjusting unit for rotating and adjusting the captured image around the optical axis of the camera may be further provided so as to have a predetermined angle.

Preferably, the finger shape detection method according to the present invention is a detection method for detecting a finger shape by using the camera assembly described above together with at least one information processing apparatus, and the information processing apparatus is used to detect the finger shape from the camera. Various CG images in which the dimensional ratio of each part of the finger and the dimensional ratio of the back of the hand are different and the pose of the finger is also different depending on the step of storing the image data obtained by capturing the image data of each finger shape to be input and the computer graphics (CG) program. A plurality of virtual finger images in which the steps for forming a finger image and the various formed finger images are the same with reference to the dimensional ratio of each part of the finger and the dimensional ratio of the back of the hand, but the poses of the fingers are different. A step of classifying into the same group and storing the dimension ratio data and the angle data of each finger joint in the pose as a data set including at least for each virtual finger image, and the dimensions of each part of the finger of the captured finger image. The ratio and the dimensional ratio of the back of the hand are discriminated, and the plurality of virtual hand images of any group are read out from the plurality of virtual finger images classified into each group and stored together with the dimensional ratio data. A step of determining whether or not the data is based on the dimensional ratio data, a step of forming a silhouette image having a contour line for both the stored image of the captured hand image and the stored virtual hand image, and the step of forming the silhouette image. A step of forming a ridge line image from any of the captured finger image, the virtual finger image, or the silhouette image, a step of superimposing the silhouette image and the ridge line image to form a superimposed image, and the superimposed image. A step of determining whether all the ridge lines of the ridge line image are contained in the internal range of the silhouette image portion, or whether there is the ridge line protruding from the external range of the silhouette image portion, and the silhouette image. When there is the ridge line that extends beyond the external range of the portion, the step of excluding the virtual finger image in the original image of the image included in the superimposed image from the candidate image of the estimation processing in the subsequent stage and the narrowing down of the candidate image. For each virtual hand image of the candidate image narrowed down by the unit, the similarity with the input imaged hand image is calculated, the virtual hand image with the minimum similarity is determined, and the virtual with the minimum similarity is determined. A step of performing an estimation process including the similarity calculation by selecting at least data including angle data for each finger joint from the data set including the finger image. At least a step of outputting the result of the estimation process to the device to be controlled is performed.

Preferably, in the finger shape detection method according to the present invention, the information processing apparatus captures the image data of each finger shape stored in the information processing apparatus, and the fingers and the back of the hand in the captured image frame. The step of enlarging or reducing the total vertical dimension of the image frame so as to substantially match the vertical dimension of the image frame, and the tilt angle of the imaged image frame captured by the information processing device are stored in the storage device in advance. The step of rotating and adjusting the captured image around the optical axis of the camera may be further performed so that the angle matches the image frame to be collated.

In order to solve the above problems, the program of the finger shape detection method according to the present invention implements the detection method described above. Further, the storage medium of the program according to the present invention stores the program described above.

According to the finger shape detection method of the present invention, a device capable of estimating all joint angles of fingers for each imaging frame by inputting captured image data from one small camera mounted on the back side of the hand is provided, and further for collation. The number of poses of finger images in the database is reduced, the reduction / enlargement ratio is reduced to reduce the amount of data, and the hand morph is effective in acquiring data at the same time as motion capture of the entire body, and further creating a collation database. A finger shape detection method that improves efficiency, reduces the time and labor required for creation, further suppresses individual differences, improves collation accuracy by suppressing noise, and further improves the identification of irregular shapes on the same contour line. And the system can be provided.

It is a block diagram which shows the schematic structure of the system which detects the shape of a finger which concerns on embodiment of this invention. It is an operation flowchart which concerns on the advance preparation of the Embodiment of this invention. It is an operation flowchart which concerns on the detection of embodiment of this invention. It is a perspective view of the camera assembly of this invention. It is a side view of the camera assembly of this invention. It is a rear view of the camera assembly of this invention. It is a figure which showed the outline of the general relationship between the input image at the time of detection and a database image. It is a figure which showed the example of the input image at the time of detection and the CG image of a database. It is a figure which showed the example of the grouping of a database. It is a figure which showed the example of the group judgment of the database from the finger length of the input image at the time of detection. It is a figure which showed the example of the group judgment of the database from the finger angle of the input image at the time of detection. It is a figure which showed the example of the data set in a database. It is the figure which showed the example which narrowed down by the out-of-range ridge line by the ridge line of an input image and the silhouette of a database. It is the figure which showed the selection example which narrowed down by the out-of-range ridge line by the ridge line of an input image and the silhouette of a database. It is the figure which showed the example which narrowed down by the out-of-range ridge line with the silhouette of an input image and the ridge line of a database. It is the figure which showed the selection example which narrowed down by the out-of-range ridge line by the silhouette of the input image and the ridge line of a database. It is a figure which showed the example of the contrast relation between each image based on an input image, and a data set of a database. It is a figure which shows the example of the relationship between the input image and CG angle data output.

<Embodiment>
Motion capture has come to be widely used for computer animation of the human body in movies and the like, and for reproducing human-like movements of characters in games. In addition, a system that draws facial expressions as computer animation in real time in combination with motion capture of human body movements has come into practical use. The last technology that is indispensable for video editing of such human body movements but has not yet reached the level of practical use is "motion capture of fine finger movements", that is, three-dimensional hand shape estimation technology (hand mocap).

In order for the finger shape estimation system to be used in combination with the motion capture system, specifications that do not interfere with motion capture are required. For example, in a large studio for optical motion capture, the shape of a relatively small finger can be estimated even if the subject moves around freely. The shape of the fingers can be estimated even if the photographed person holds something in his hand or his palm is brought into contact with another object such as the floor, wall, desk, or another person, and the device is less likely to cause fatigue to the photographed person. And the weight.

One method is to use a wireless data glove (for example, CyberGlove II manufactured by CyberGlove Systems Inc.). However, data gloves are expensive, have a strong sense of restraint, and the strain gauges and suspension lines (wiring) that are sensors are easily broken, so they cannot be used for motion capture of violent movements. Also, it is not suitable for manipulating objects with the palm or gripping to form a fist.

Alternatively, Digits is an example of wireless contact finger shape estimation. An infrared sensor or an infrared camera is attached to the wrist, and the finger is photographed from the palm side to measure the distance from the sensor to the finger and estimate the shape. However, since the system attaches a sensor to the wrist, it is difficult to respond when the wrist is bent and stretched. In addition, since the mounting position is on the palm side, it interferes with the wearer in daily activities such as walking, gripping, operating, and touching other objects, and the measurement itself cannot be performed.

Therefore, the inventors have proposed a compact system in which an ultra-small wireless RGB camera is attached to the back side of the hand instead of the palm side to estimate the shape of the fingers. We also implemented an actual machine that combines an ultra-compact RGB camera and a single board computer. By installing the camera on the back side of the hand, it can be expected that the restraint of the subject's movement in motion capture will be minimized. The reason why the conventional method such as Reference [1] installed the camera on the palm side was that the fingertips had to be imaged at all times, and that shielding was unlikely to occur even if the knuckle was bent. However, this method has proposed an algorithm that can estimate the shape of a finger even if the fingertip or a part with a finger is shielded.

However, the methods proposed above include, in particular, "finger length, thickness, palm width, thumb base position, four finger opening degree" or "range of motion of each finger and within the range of motion. It was not possible to estimate the shape of the fingers with high accuracy according to the individual differences of the user, such as "the habit of moving with." In general, if the number of collation databases is increased, the estimation resolution can be increased, but the processing time is increased. It is also not preferable to make the calibration time before estimation too long. Therefore, in the present invention, a wearable finger shape estimation device characterized by the following is devised:

1. 1. Prepare multiple types of collation database (CG hand) groups in advance with different parameters that characterize the shape of the finger, such as finger length and thickness, and the position of the base of the thumb, and select the user's finger from among them before estimation. By selecting the most similar database group and performing similarity matching between the matching data set in that group and the input image.
There are individual differences (for example, thick fingers, short fingers, bent thumbs, the base position of the thumb is close to the wrist, the palm is wide, the distance between the two fingers is wide when the V sign is made with the index finger and the middle finger, etc. ) The shape of the finger can be estimated with high accuracy.

1-2. When creating a hand database for collation
In particular, by creating a collation data set in which the finger thickness, length, palm width, and thumb base position ratio are different, a collation database capable of high-precision estimation can be efficiently created.

1-3. When selecting a collation database group that most closely resembles the user's finger
In particular, in the finger image of the user and the finger image of the collation data set, the comparison and determination are made by paying attention to the ratio of the finger thickness, the length, the width of the palm, and the position of the base of the thumb.

2-1. After selecting the collation database, the finger shape can be estimated with higher accuracy.
When fine-tuning the camera position after mounting the device, when fine-tuning the distance between the camera and the user's fingers,
Fine-tune the camera position so that the silhouette of the collation dataset base and the input image match best. Specifically, the camera position is moved back and forth so that the length from the base to the tip of each of the five fingers is most suitable.

2-2. When fine-tuning the camera position after mounting the device, if you want to fine-tune the tilt of the camera,
Make sure that the tilt of the fingers matches best between the collation dataset base and the input image.
For example, rotate the camera so that the two middle fingers are tilted.

3-1. When comparing the input image with the collation database image, if there is no finger in the area where it should be in the image, or conversely, there is a finger in the area where it should not be, a similarity matching method that gives a penalty As the estimation result, the collation data set significantly different from the input image is not output.

3-2. Image processing by collating the ridge line image with either the input image or the collation database image thinned with the original finger image without thinning, and using the protruding part or the protruding size as a penalty. Reduces the effects of time noise and individual differences in finger shape.

The hardware of the camera assembly consists of a support column 31 made of flexible wire, a wrist supporter, and a mobile battery. The finger image taken by the wireless camera is transmitted to the computer to estimate the finger shape. 4 to 6 show the appearance of the device. The wireless camera used was a 5.8G Wireless Mini Camera -TE60A manufactured by 3rd Eye Electronics, which is a small RGB camera. The angle of view is 90 °, and the resolution of the acquired image is 720 x 480 [pixel]. As shown in FIGS. 4 to 6, the position of the camera 35, the hand-back portion 51 side, is in between the thumb 54 and index finger 55, hand-back direction from the thumb 54 side little finger 58 side, also, from the fingertip 52 end Install it at a slight angle toward the section 51 side. Length of the camera 35 and the hand-back portion 51 is about 120 [mm]. The power cable 36 connecting the camera 35 and the battery 38 is longer than the arm length, and the battery 38 is attached near the trunk where no moment is felt. The weight of the camera assembly excluding the battery 38 is about 75 [g].

<Construction of hand database for collation>
The collation database builds finger joint angle information, finger CG silhouette information created by CG editing software based on it, image feature quantities (for example, higher-order local autocorrelation), and ridge line information as one data set. To do. A collation database is a collection of data sets for various appearances of various finger shapes. The structure of the collation database is as shown in FIG.
At the time of estimating the shape of the finger, the input finger image is collated with the collation finger database by using the silhouette information, the image feature amount, and the ridge line information. When the finger data set most similar to the input image is selected as a result of the collation, the finger joint angle information of the data set is output as the estimation result. (The joint angle information is the information used to create the finger CG).

<Fine adjustment of camera position>
In order to select a finger data set that is as similar as possible to the finger image captured and input by the camera, the position of the camera attached to the back of the hand is finely adjusted prior to the actual estimation. Fine adjustment is performed in the following two steps.

[Step 1] In order to fine-tune the "distance" between the camera and the user's finger, the camera position is fine-tuned so that the silhouette of the collation data set base and the input image match best. Specifically, as illustrated in FIG. 11, the camera position is moved back and forth so that the lengths from the base to the tip of each of the five fingers are most suitable.

[Step 2] In order to fine-tune the "tilt" of the camera, the camera is rotated so that the tilt of the fingers matches the collation data set base and the input image most. For example, as illustrated in FIG. 12, the camera is rotated so that the inclinations of both middle fingers match.

<Estimation flow>
The user wears the proposal system and calculates HLAC features, ridge line information, and silhouette information from the input image. The database is searched based on the calculated features, and the data set most similar to the input image is used as the estimation result. Finally, the finger joint angle of the estimation result data set is output. The details will be described below.

<Narrowing down the search target>
In order to exclude the data set that has a posture significantly different from the input image from the search target, the search target is narrowed down in two stages. The penalty described later is calculated using the acquired silhouette information and ridge line information.

The idea here is to exclude the collation data set that "should not be" in the input image and "not" in the input image, as illustrated in FIGS. 13 and 14, and conversely, the finger that "should not be" in the input image. This is because the two-way exclusion process of excluding the collation data set that "exists" is performed.

First, a process is performed to exclude the collation data set that does not have a finger that "should exist" in the input image from the search target. As shown in FIGS. 13 and 14, the area where the ridge line area of the input image protrudes from the silhouette area of the database image is calculated based on the ridge line information of the input image and the silhouette information of the data set, and the protruding area is calculated. the value [pixel] and penalty P _1. Then, as the narrowing down of the first stage, the data set in which the value of P ₁ exceeds the threshold value shown by the following formula is excluded from the search target.

After narrowing down in the first stage, processing is performed to exclude the collation data set that has a finger "should not be" in the input image from the search target among the collation data sets remaining as the search target. As shown in FIGS. 15 and 16, the area where the ridge line area of the data set protrudes from the silhouette area of the input image is calculated based on the ridge line information of the data set and the silhouette information of the input image, and the protruding area is calculated. the value [pixel] and the second penalty P _2. Then, as the second stage narrowing down, the data set in which the value of P ₂ exceeds the threshold value th ₂ shown by the following equation is excluded from the search target.

<Similarity calculation>
After narrowing down the search target by the method shown above, the search is performed by the similarity calculation for the collation data set remaining as the estimation target. The similarity, using the penalty P _1, P ₂ shown in HLAC features and above is calculated by the following equation.

As a result of searching all the databases, the data set with the smallest similarity obtained by the similarity calculation is used as the estimation result, and the finger joint angle of the estimation result data set is output.

<System configuration>
The system for detecting the shape of a finger according to the present embodiment of FIG. 1 includes an information processing device 1, a camera 35, and a display device 100, and the information processing device 1 is communicatively connected to the camera 35 and the display device 100.

The finger / back various dimension data storage unit 21 stores various finger / back dimensions corresponding to different finger shapes and angle data of each joint, and can read and output the data. CG images of various poses are formed by the CG finger image forming unit 23 corresponding to each data, grouped by the group control unit 24 according to various dimensions of the fingers / back of the hand, and collated CG fingers in the information processing device 1. Data corresponding to the same CG image and processed images are associated with each group in a set in the image database 25 and stored in the collation CG hand image database 25 in the information processing apparatus 1.

As the display device 100, a flat display such as a normal LCD can be used for applications such as confirmation of the input image and / or the shape of the finger detected from the input image and confirmation of the contour line. In addition, the display device 100 can display the reproduced or synthesized finger shape based on the finger shape detected from the input image.

The information processing device 1 calculates the image shape ratio data and the image feature amount data including the brightness gradient direction vector from the image data obtained by capturing each finger shape input from the camera 35, and both data are hand morphs of the device mounting method. It is stored in the collation database corresponding to the data sets of the plurality of finger shapes whose shapes are detected by.

In the information processing device 1, an image dimension adjustment unit 11, an image angle adjustment unit 12, an image pickup finger image storage unit 13, a silhouette image formation unit 14, a ridge line image formation unit 15, a contour line extraction unit 16, and an image feature amount detection. Unit 17, superimposed image forming unit 18, out-of-range ridge line determination unit 19, candidate image narrowing unit 20, finger / back various dimension data storage unit 21, joint angle storage unit 22, CG finger image forming unit 23, group control unit 24. , CG hand image database 25, image feature amount similarity calculation unit 26, group determination unit 27, finger angle estimation unit 28, data output unit 29, various set value storage units 71, program storage unit 81, control unit 91, and display. The device 100 is provided and is communicably connected from the camera 35 side toward the display device 100 side.

The CG finger image database 25 is formed by the CG finger image forming unit 23 together with the image data of each frame captured by the camera 35 and its processed data (silhouette image, contour line image, ridge line image, image feature amount). As for the image, the CG image data, its processing data (silhouette image, contour line image, ridge line image, image feature amount), joint angle data, and the like are stored.

The image size adjusting unit 11 adjusts the size of the image input from the camera 35 to a size suitable for the detection process. This can be adjusted on the camera mounting portion 33 side of the camera 35, but the image dimension adjusting portion 11 is used when it cannot be handled by itself or when the camera 35 side does not support it.

The image angle adjusting unit 12 adjusts the angle of the image input from the camera 35 to an angle suitable for the detection process. This can be adjusted on the camera mounting portion 33 side of the camera 35, but the image angle adjusting portion 12 is used when it cannot be handled by itself or when the camera 35 side does not support it.

The imaging finger image storage unit 13 stores the image input from the camera 35 and outputs it to the silhouette image forming unit 14 and the group determination unit 27 in the subsequent stage.

The silhouette image forming unit 14 forms a silhouette image from the captured image data, the ridge line image forming unit 15 forms a ridge line image from the silhouette image, and the contour line extracting unit 16 forms a contour line image from the silhouette image. Then, the image feature amount detection unit 17 detects the image feature amount from the contour line image by the divided region. Each image and feature amount corresponding to the captured image data for each frame is stored in the CG finger image database 25.

On the other hand, with respect to the CG finger image formed by the CG finger image forming unit 23, the sill silhouette image forming unit 14 forms a silhouette image from the CG finger image data, and the ridge line image forming unit 15 forms a ridge line image from the silhouette image. Is formed, the contour line extraction unit 16 forms a contour line image from the silhouette image, and the image feature amount detection unit 17 detects the image feature amount from the contour line image by the divided region. Each image corresponding to the CG finger image data, the feature amount, and the joint angle data are stored in the CG finger image database 25 as set data.

In the superimposed image forming unit 18, the silhouette image and the ridge line image corresponding to the captured image data for each frame are read from the CG finger image database 25, and the ridge line image and the silhouette image of the set data in the CG finger image database 25 are combined. They are superimposed in correspondence with each other. The out-of-range ridge line determination unit 19 detects and determines the ridge line protruding from the silhouette image for each superimposed image. In the candidate image narrowing unit 20, the CG finger image whose ridge line protrudes from the superimposed image is excluded from the estimation candidates and narrowed down.

The finger / back various size data storage unit 21 stores various data of the finger / back for forming a finger image by CG. The joint angle storage unit 22 in the joint angle storage unit 22 stores angle data for each joint. A CG finger image is formed in the CG finger image forming unit 23 based on each data. Each CG finger image is group-classified by the group control unit 24 according to the same finger / back size and stored in the CG finger image database 25.

The image feature amount similarity calculation unit 26 calculates the similarity from the image feature amount of the input captured image and the image feature amount of each CG finger image and transmits the similarity to the narrowing down unit 20.

The finger angle estimation unit 28 estimates the most similar image from each CG finger image of the narrowed down group, sends the angular joint angle data to the data output unit 29, and from there, the external device (display device 100). Send to.
The various set value storage units 71 store various set values of the information processing device 1, the program storage unit 81 stores a program for operating each of the above units, and the control unit 91 controls each of the above units.
Control unit 91 and display device 100

The operation of the system for detecting the shape of the fingers according to the present embodiment will be described with reference to the flowchart of FIG. First, a collation database is constructed before the actual finger shape detection is performed. The information processing device 1 forms a CG finger image from the stored angle / finger / back data and the like (S51). The formed finger images are group-classified by finger size group (52). The finger images of each group are stored in the database 25 (S53).

A silhouette image is formed from the CG finger image and stored in the database 25 (S54), a ridge line image is formed from the silhouette image and stored in the database 25 (S55), and a contour line image is formed from the silhouette image and stored in the database 25. (S56), the image feature amount is detected from the contour line image and stored in the database 25 (S57), and each image having the same original CG image, the image feature amount, and the angle data are stored in the database 25 as set data. Remember (S58).

In step S1, the dimensions of the captured finger image are adjusted (S1), then the angle of the hand image is adjusted (S2), and the captured finger image is stored (S3). Next, a silhouette image is formed from the captured finger image (S4), and a group is determined from the captured image and the silhouette image (S5). The data set of the determined group is extracted from the database (S6).

A ridge line image is formed from the silhouette image of the captured image (S7). The silhouette image of the data set and the formed ridge line image are superimposed (S8). An image in which the ridge line is outside the silhouette range is determined (S9). A CG finger image whose ridge line of the superimposed image is outside the silhouette range is excluded from the candidates for estimation processing (S10). The image feature amount is detected from the silhouette image of the finger image (S11). The degree of similarity is collated with the image feature amount of the collation database (S12). It is determined whether or not there is the next image data in the group (S13). If there is next data (S13: NO), the process returns to step S7 to perform processing on the next image.

When there is no next data (S13: YES), the finger angle of the most similar CG finger image is read from the data set of the database (S14), and the most similar finger angle is output (S15).

As described above, according to the finger shape detection method of the present embodiment, a device capable of estimating all the joint angles of the fingers for each imaging frame by inputting captured image data from one small camera attached to the back side of the hand is provided. In addition, the number of poses of the finger image in the collation database is reduced, the reduction / enlargement ratio is reduced to reduce the amount of data, and the hand morph is effective in acquiring data at the same time as motion capture of the entire body, and further collation. The time and effort required to create a database can be streamlined, individual differences can be further suppressed, and noise can be suppressed to improve collation accuracy and further improve the identification of irregular shapes with the same contour line. A method and system for detecting finger shape can be provided.

1 Information processing device,
11 Image dimension adjustment unit,
12 Image angle adjustment unit,
13 Imaging finger image storage unit,
14 Silhouette image forming part,
15 Ridge line image forming part,
16 Contour line extractor,
17 Image feature detection unit,
18 Superimposed image forming part,
19 Out-of-range ridge line judgment unit,
20 Candidate image narrowing section,
21 Finger / back of hand various size data storage unit,
22 Joint angle memory,
23 Virtual ( CG ) finger image forming part,
24 group control unit,
25 virtual ( CG ) finger image database,
26 Image feature similarity calculation unit,
27 Group Determination section,
28 Finger angle estimation unit,
29 Data output section,
30 Support,
31 Support pillar,
32 Foundation,
33 Camera mounting part,
34 Back fixing part,
35 camera,
36 power cable,
37 wireless antenna,
38 battery,
51 back of the hand,
52 fingers,
53 Transversal line at the base of fingers,
54 thumbs,
55 index finger,
56 middle finger,
57 Ring finger,
58 little finger,
71 Various setting value storage units,
81 Program storage,
91 Control unit,
100 display device,
θ1 forward tilt angle,
θ2 Lateral thumb side tilt angle.

Claims (12)

One or more small cameras are supported in the sky near the back of one hand of the wearer so that substantially the entire one hand of the wearer is contained within the angle of view of the camera, and the input captured by the camera. A camera assembly that outputs images and
With at least one information processing device
It is a finger shape detection system that detects the shape of a finger .
The information processing device
An imaging hand image storage unit that stores image data of an input image that captures each finger shape input from the camera,
A virtual finger image forming unit that forms virtual finger images of various CG images in which the dimensional ratio of each part of the fingers and the dimensional ratio of the back of the hand are different depending on the computer graphics (CG) program, and the poses of the fingers are also different.
A virtual finger image database that stores angle data for each finger joint and image feature data calculated from the image data as a data set that includes at least each virtual finger image .
Before SL and captured hand image stored in the captured hand image storage unit, for both the image and the virtual hand image stored in the virtual finger image database, and the silhouette image forming unit for forming a silhouette image having a contour,
A ridge line image forming unit that forms a ridge line image from any of the captured finger image, the virtual finger image, or the silhouette image.
A superimposed image forming unit that superimposes the silhouette image and the ridge line image to form a superimposed image,
An out-of-range ridge for determining whether all the ridge lines of the ridge line image are contained in the internal range of the silhouette image portion of the superimposed image, or whether the ridge line extends beyond the external range of the silhouette image portion. Line judgment part and
When the ridge line extends beyond the external range of the silhouette image portion, the virtual finger image in the original image of the image included in the superimposed image is excluded from the candidate images used in the subsequent estimation process. Narrowing down part and
At least have a,
Exclusion by the candidate image narrowing section
Using the ridge line information of the input image and the silhouette information of the data set, the ridge line region of the input image protrudes from the internal range of the silhouette image portion of the virtual finger image stored in the virtual finger image database. Calculate the area of
The value of the first area that protrudes is used as the value of the first penalty, and the input image "should" have a virtual finger image of a data set in which the value of the first penalty exceeds the first predetermined threshold value. Exclude from the search target of the virtual finger image that is a candidate of the virtual finger image database as a virtual finger image that has no finger.
Finger shape detection system. The exclusion by the candidate image narrowing section further
Using the ridge line information of the input image and the silhouette information of the data set, the ridge line region of the data set of the virtual finger image stored in the virtual finger image database is from the internal range of the silhouette image portion of the input image. Calculate the second area that sticks out,
The value of the second area that protrudes is used as the value of the second penalty, and the virtual finger image of the data set in which the value of the second penalty exceeds the second predetermined threshold is "should not be" in the input image. Exclude the virtual finger image that has a finger from the search target of the virtual finger image that is a candidate of the virtual finger image database.
The finger shape detection system according to claim 1. Exclusion by the candidate image narrowing section
The input image "should" have a virtual finger image of a dataset in which the value of the first penalty exceeds the first predetermined threshold and the value of the second penalty exceeds the second predetermined threshold. Exclude from the search target of the virtual finger image that is a candidate of the virtual finger image database as a virtual finger image that does not have a finger and has a finger that "should not exist".
The finger shape detection system according to claim 2. Further, the similarity is calculated from the value of the first penalty, the value of the second penalty, and the data of the image feature amount, and the finger angle estimation using the data set with the smallest obtained similarity as the estimation result is used. Department and
A data output unit that outputs the finger joint angle of the estimation result data set, and
The finger shape detection system according to claim 2 or 3. With respect to the image data of each finger shape stored in the image-imaging finger image storage unit, the total vertical dimensions of the fingers and the back of the hand in the captured image frame substantially match the vertical dimensions of the image frame. An image dimension adjustment unit that enlarges or reduces,
The captured image is rotated around the optical axis of the camera so that the tilt angle of the captured image frame is a predetermined angle that matches the image frame to be collated stored in the storage device in advance. The image angle adjustment unit to be adjusted and
The finger shape detection system according to any one of claims 1 to 4 . One or more small cameras are supported in the sky near the back of one hand of the wearer so that substantially the entire one hand of the wearer is contained within the angle of view of the camera, and the input captured by the camera. A camera assembly that outputs images and
With at least one information processing device
It is a finger shape detection method that detects the finger shape using
A step of storing the image data of the input image obtained by capturing each finger shape input from the camera, and
In the information processing apparatus,
A step different dimensional ratios of various dimensions ratios and hand-back of the fingers, and, for forming a virtual finger image poses also different various CG image finger by your computer graphics (CG) program,
A step of storing the angle data for each knuckle joint and the data of the image feature amount calculated from the image data as a data set including at least for each virtual finger image .
An imaging hand image stored before reporting, for both image and the stored virtual finger image, forming a silhouette image having a contour,
A step of forming a ridge line image from any of the captured finger image, the virtual finger image, or the silhouette image.
A step of superimposing the silhouette image and the ridge line image to form a superimposed image,
A step of determining whether all the ridge lines of the ridge line image are contained in the internal range of the silhouette image portion of the superimposed image, or whether the ridge line extends beyond the external range of the silhouette image portion.
When there is the ridge line that extends beyond the external range of the silhouette image portion, the step of excluding the virtual finger image in the original image of the image included in the superimposed image from the candidate image of the estimation processing in the subsequent stage, and
At least carry out,
The excluded step is
Using the ridge line information of the input image and the silhouette information of the data set, the ridge line region of the input image protrudes from the internal range of the silhouette image portion of the virtual finger image stored in the virtual finger image database. Calculate the area of
The value of the first area that protrudes is used as the value of the first penalty, and the input image "should" have a virtual finger image of a data set in which the value of the first penalty exceeds the first predetermined threshold value. Exclude from the search target of the virtual finger image that is a candidate of the virtual finger image database as a virtual finger image that has no finger.
Finger shape detection method. The exclusion step further
Using the ridge line information of the input image and the silhouette information of the data set, the ridge line region of the data set of the virtual finger image stored in the virtual finger image database is from the internal range of the silhouette image portion of the input image. Calculate the second area that sticks out,
The value of the second area that protrudes is used as the value of the second penalty, and the virtual finger image of the data set in which the value of the second penalty exceeds the second predetermined threshold is "should not be" in the input image. Exclude the virtual finger image that has a finger from the search target of the virtual finger image that is a candidate of the virtual finger image database.
The finger shape detection method according to claim 6. The excluded step is
The input image "should" have a virtual finger image of a dataset in which the value of the first penalty exceeds the first predetermined threshold and the value of the second penalty exceeds the second predetermined threshold. Exclude from the search target of the virtual finger image that is a candidate of the virtual finger image database as a virtual finger image that does not have a finger and has a finger that "should not exist".
The method for detecting the shape of a finger according to claim 7. Further, a step of calculating the similarity from the value of the first penalty, the value of the second penalty, and the data of the image feature amount, and using the data set with the minimum obtained similarity as the estimation result. ,
A step of outputting the finger joint angle of the estimation result data set,
7. The finger shape detection method according to claim 7 or 8. With the information processing device
With respect to the image data of each finger shape stored in the information processing device, the total vertical dimension of the finger and the back of the hand in the captured image frame is enlarged or enlarged so as to substantially match the vertical dimension of the image frame. Steps to shrink and
In the information processing device, the captured image of the camera is set so that the tilt angle of the captured image frame is a predetermined angle that matches the image frame to be collated stored in the storage device in advance. Steps to adjust the rotation around the optical axis,
The method for detecting the shape of a finger according to any one of claims 6 to 9 , further comprising the method. A program for implementing the finger shape detection method according to any one of claims 6 to 10 . A storage medium for storing the program according to claim 11 .