JP2009003813A - Finger shape estimation apparatus, finger shape estimation method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To more accurately estimate a finger shape including a wrist turn from an image of fingers captured by a camera. <P>SOLUTION: Image data corresponding to various finger shapes and nail position data representing nail positions in the shapes represented by the image data are stored to construct a database. Nail positions are detected in image data obtained by imaging fingers to be detected, and the detected nail positions are collated with the nail position data in the database. On the basis of the nail position data substantially matching the detected nail position data, the image data stored in the database in association with the nail position data is read out, and the finger shape represented by the image data is set as an estimated shape of the fingers to be detected. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a finger shape estimation apparatus and a finger shape estimation method suitable for driving and controlling a finger of a robot or a virtual object, for example, and a program for executing the detection method.

In recent years, multi-finger type robot hands having a shape similar to human fingers have been developed. The multi-fingered robot hand has a structure in which the joints move like a human hand, so that a work such as grasping an object can be performed in the same manner as a human hand, and an advanced work can be performed. As one method for controlling the operation of the robot hand, the operator's fingers are photographed with a camera, and the robot hand reproduces the same shape as that of the finger estimated from the image of the finger photographed with the camera. Has been proposed.
Japanese Patent Laid-Open No. 2004-228688 describes a technique for controlling a robot hand by estimating the shape of a finger from an image of a finger previously proposed by the inventors of the present invention. In the technique described in Patent Document 1, the contour of a finger is detected from an image obtained by photographing the finger, and the shape of the finger is estimated from the contour.
JP 2006-294018 A

  By the way, as described in Patent Document 1, it is possible to estimate the shape of a finger with a certain degree of accuracy by detecting the contour of the finger from an image obtained by photographing the finger. In the shape estimation, the angles of the joints of the fingers may be erroneously estimated. Specifically, it may be difficult to determine whether the palm or the back of the hand is visible only by the contours of the fingers. Furthermore, there are cases where it is difficult to accurately determine the angle of each finger joint from an image of a finger bent.

  The present invention has been made in view of such a point, and an object thereof is to more accurately estimate the shape of a finger from an image obtained by photographing the finger with a camera.

The present invention stores image data corresponding to various shapes of fingers and nail position data indicating the position of the nail when the shape is indicated by the image data, and forms a database.
Then, the position of the nail is detected from the image data obtained by photographing the finger to be detected, and the detected nail position is collated with the nail position data stored in the database. Further, based on the nail position data for which a match is detected by the collation, the image data stored in the database corresponding to the nail position data is read, and the finger shape indicated by the image data is changed to the detection target finger. It is determined as the shape of the.

According to the present invention, it is possible to accurately estimate a finger shape using a nail position detected from finger image data obtained by photographing. Therefore, advanced control of various devices is possible based on the finger shape estimated from the captured image.
For example, by controlling the robot hand to reproduce the estimated finger shape, the robot hand can be controlled easily and accurately from the movement of the operator's finger.
In addition, the three-dimensional CG model of the hand can be operated based on the finger shape estimated from the captured image. By adding a function such as hit determination to the three-dimensional CG model of the hand, a three-dimensional object operation can be performed in the virtual world in the computer.

  In addition, the finger nail position information and the contour information are stored together, and the angle of each joint of the finger can be accurately estimated by using both the nail position data stored in the database and the finger contour information for collation. It becomes like this.

  In this case, the joint joint angle data or three-dimensional coordinate data corresponding to the image data is stored as the database, and the joint angle data stored in the database corresponding to the nail position data in which matching is detected by the collation. Alternatively, it is possible to accurately drive the robot hand and the finger CG by estimating the joint angle of the finger from the three-dimensional coordinate data.

  In addition, the joint joint angle data or three-dimensional coordinate data stored in the database can be easily obtained by using a so-called data glove or magnetic sensor with a sensor attached to each joint of the fingers. It becomes possible to obtain three-dimensional coordinate data.

  Further, the skin color portion of the finger and the nail color portion included in the skin color portion are extracted from the image data obtained by photographing the detection target finger, and the nail color portion is extracted from the extracted nail color portion. By detecting the position, it is possible to efficiently detect the nail.

  Further, when detecting the nail color portion from the skin color portion, it is expressed by a first axis passing through the vicinity of the skin color portion of the finger to be detected and a second axis orthogonal to the first axis. By defining the color space of the image data according to the area to be extracted and extracting the nail color part from the defined skin color part, the nail color part can be accurately extracted with little estimation error, Detection accuracy is improved. Further, since there is little useless data in the skin color portion, the amount of calculation processing for extracting the nail color portion can be reduced, and the nail portion can be detected at high speed.

  Further, when setting the first axis and the second axis, the angular position of each axis of the color space set by the first and second axes is an image obtained by photographing the skin color portion of the detection subject in advance. By setting based on the data, it is possible to perform highly accurate detection that eliminates errors due to differences in skin color of individual detection subjects.

Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.
In this embodiment, a finger to be detected is photographed with a video camera, and the shape of the finger and the joint angle are estimated from the photographed image. The estimated finger shape and joint angle data are used to control various devices such as a robot hand and a three-dimensional CG model of the hand.

  First, the system configuration of the apparatus will be described with reference to FIG. As shown in FIG. 1, the apparatus according to the present embodiment includes a database device 10 that stores data such as the shape of a finger, and a hand estimation that estimates the shape of a finger while referring to the data stored in the database device 10. The apparatus 20 is comprised. Although the database device 10 and the finger estimation device 20 are configured as separate devices in FIG. 1, they may be configured integrally.

The database device 10 is means for storing data on various shapes of fingers. In the case of this example, image data captured by the video camera 31 and data of each sensor attached to the data glove 40 are supplied. The video camera 31 captures an image of a finger for a database. The data glove 40 is a glove worn on a finger when the image is taken.
FIG. 2 shows an example of the data glove 40, and a sensor 41 for detecting an angle at which the joint moves is arranged at a position corresponding to each joint of the finger. In order to detect wrist movement, a sensor 41 is also arranged on the palm. In the following description, the movement of the wrist is also included when the movement of the finger joint is described. As the sensor 41, for example, a strain sensor is used. Further, in the data glove 40 of this example, the nail 42 is written at a position corresponding to the nail of each finger with a color (for example, red) different from the color (for example, white) of the data glove itself.

Returning to the description of FIG. 1, the image data captured by the video camera 31 is supplied to the image data storage means 11 and temporarily stored. The stored image data is supplied to the image feature quantity extraction means 12 and a feature quantity extraction process is performed to extract the contours of fingers in the image. The feature amount extraction processing will be described later.
The nail position calculation means 13 performs processing for detecting the position of the fingernail in the image. The details of the nail position detection processing here will be described later. Briefly, the nail color written in the data glove 40 is detected, and the detected portion is used as the nail position. . The position of the nail is, for example, data on the position of the center of gravity of the nail.
The feature amount data detected by the image feature amount extraction unit 12 and the nail position data detected by the nail position calculation unit 13 are supplied to the image and nail position corresponding operation command storage unit 14 and stored in a database. Let

  Further, the output of each sensor of the data glove 40 is periodically sampled by the sampling means 15, and the sampled sensor output is supplied to the time series joint angle data storage means 16 to obtain time series joint angle data. Remember. The stored time-series joint angle data is also supplied to the image and nail position corresponding operation command storage means 14 and stored in a database.

When creating a database in the image and nail position corresponding operation command storage means 14, the finger shape data, the nail position at the time of the shape, and the angle data of each joint of the finger at the shape, Create a database in association.
The database stored in the storage unit 14 is read by an instruction from the finger shape estimation device 20.

  The finger shape estimation device 20 supplies finger image data captured by the video camera 32 to the personal color system conversion means 21 and converts the coordinate axes of the color system of the image data with predetermined conversion characteristics. . Details of the conversion process will be described later. The personal color system conversion means 21 stores conversion characteristics as initial settings when the operator's fingers are photographed by the video camera 32.

The image data converted by the personal color system conversion means 21 is stored in the image data storage means 23, the image feature quantity is extracted by the image feature quantity extraction means 24 from the stored image data, and the contour of the finger is extracted. Is done. Further, the nail position calculating unit 25 detects the position of the nail from the image data stored in the image data storage unit 23.
The feature amount data extracted by the image feature amount extraction unit 24 and the nail position data detected by the nail position calculation unit 25 are sent to the data specification and operation command generation unit 26. The data specifying and operation command generating means 26 searches for similar data on the database stored in the database device 10 from the supplied feature value data and nail position data, and corresponds to the closest data. The stored joint angle data is read from the database.

Then, the joint angle data read from the database is estimated as the angle of each joint of the photographed finger. The estimated joint angle data is generated as an operation command for each joint of the robot hand, the operation command is sent to the driving means 27 of the robot hand, and the actuator of each joint in the robot hand 50 is set in a corresponding state. Drive.
In the configuration of FIG. 1, the video camera 31 for database construction and the video camera 32 used for actually estimating the shape of the fingers are different from each other. Good. The video cameras 31 and 32 are preferably those capable of high-speed shooting at about 150 frames per second, for example, but normal video cameras that perform shooting at about 60 frames per second may be used.

FIG. 3 shows an example of the robot hand 50 that is driven based on the finger shape estimated by the finger shape storage means 20 of this example.
As shown in FIG. 3, the robot hand 50 has a configuration in which a joint actuator 52 is arranged at a position corresponding to an actual finger joint position and is supported by a support member 51. An actuator (not shown) is also provided in the support member 51, and the angle at which the robot hand is supported by the support member 51 can be adjusted in accordance with the state where the wrist is bent. Each joint actuator 52 is driven so as to be set to a corresponding angle by the data of each joint angle read from the database.

FIG. 4 shows an example of creating a three-dimensional CG based on the finger shape estimated by the finger shape storage means 20 of this example and displaying a virtual object in the virtual world in the computer operated by the finger of the estimated shape. Is shown. A virtual object is an object whose shape is freely deformed depending on how to hold or touch it.
As shown in FIG. 4, a camera-equipped HMD (head mounted display) 90 is mounted on the operator's head, and the operator superimposes the image displayed on the camera-equipped HMD 90 on the surrounding real image. Can see. A 3-axis position sensor 91 is attached to the camera-equipped HMD 90, and the position information of the head is acquired by the output of the 3-axis sensor 91, and the position where the operator is viewing the virtual object is matched. . The camera attached to the camera-equipped HMD 90 is configured to photograph the operator's front hand and photographs the operator's right hand HR and left hand HL. In FIG. 4, a three-dimensional virtual object 92 is shown, but this three-dimensional virtual object 92 is visible only to an operator wearing the HMD 90 with a camera.

By photographing the right hand HR and the left hand HL of the operator with the camera in the HMD 90 with a camera, the hand shape storage means 20 of this example estimates the shape of the finger from the photographed right hand HR and left hand HL image data. . A three-dimensional CG of the hand is created based on the estimated finger shape. A three-dimensional CG is created from CG data.
FIG. 5A shows the right hand 93R and the left hand 93L by the created three-dimensional CG. The right hand 93R and the left hand 93L created by the three-dimensional CG are not displayed in this example, and are used to determine the shape of the three-dimensional object 92 to be operated by holding the hands 93R and 93L of the three-dimensional CG. Is done.

FIG. 5B is an example of an image that can be seen by an operator wearing the HMD 90 with a camera. In this example, the operator has the three-dimensional virtual object 92 with the actual right hand HR and left hand HL. The three-dimensional virtual object 92 is deformed so as to be operated with the right hand HR and the left hand HL based on the estimation results of the shapes of the right hand HR and the left hand HL. For example, when the right hand HR and the left hand HL are separated from each other, the three-dimensional virtual object 92 is stretched, and when the right hand HR and the left hand HL are brought close to each other, the three-dimensional virtual object 92 is reduced. By performing the display in this manner, the three-dimensional virtual object is manipulated and deformed as if it were a real world object according to the finger movement of the operator.
Further, as shown in FIG. 5B, as the image display by the HMD 90 with a camera, a menu display 94 is performed, and an operation in which each item in the menu display 94 is touched by a hand is performed, whereby the virtual object 92 is colored. Various image operations such as attaching may be performed.

By constructing a device that performs such display processing as an OS (operating system) as a desktop manager, it becomes an OS that can be operated three-dimensionally.
In the example of FIG. 5B, the three-dimensional virtual object 92 is displayed so as to overlap the actual right hand HR and left hand HL. However, the right hand 93R and the three-dimensional CG shown in FIG. The left hand 93 and the three-dimensional virtual object 92 may be displayed.

Next, the general flow of the finger determination process according to the present embodiment will be described with reference to the flowcharts of FIGS.
First, the flow of processing for constructing a database by the database device will be described with reference to FIG.
When constructing the database, first, an image of a finger wearing the data glove 40 shown in FIG. 2 is captured, the shape of the finger and the nail position are detected from the image of the finger, and the output of the sensor 41 of the data glove 40 at that time Is converted into a joint angle and read (step S101). Then, the read finger shape, nail position, and joint angle are associated with each other to create a database (step S102). After that, it is determined whether or not to continue the database construction (step S103), and when the database construction is continued, the process returns to step S101, and another shape (that is, a shape in which the degree of bending of the finger joint is different). ). In this way, various finger shapes are made into a database, and when it is determined in step S103 that the database construction is finished, this database construction processing is finished.

  Note that the finger joint angle data at the time of database construction does not necessarily need to use a sensor such as a data glove, and a finger CG model generated by inputting an arbitrary joint angle may be used. Further, the joint angle data of fingers at the time of database construction may be three-dimensional coordinate data. It goes without saying that a magnetic sensor, a color glove, or the like can be used as the sensor in that case.

Once the database is prepared, the initial adjustment is performed by photographing the fingers of the person to be detected. FIG. 7 is a flowchart showing the flow of the initial adjustment process. In the initial adjustment process, the video camera 32 is used to photograph the finger of the person to be detected, and the photographed image is read as a reference image (step S111). The reference image, which is an image of a finger, is preferably a plurality of images, for example, and at least one of the plurality of images preferably includes a nail.
Based on the read reference image, it is determined in which range the skin color space of the finger exists, and based on the determination, the axis of the color space necessary for detecting the finger of the detection target person The conversion characteristics are calculated (step S112). The calculated conversion characteristics are stored in, for example, the personal color data storage unit 22 shown in FIG. 1, and the initial adjustment process ends.
When the initial adjustment is completed, it is possible to perform processing for estimating the shape by photographing the finger of the detection target person. Note that this initial adjustment may be performed by reading the stored data if the conversion characteristics based on the finger of the person to be detected this time have already been stored in the apparatus. It's not an easy process.

The flowchart in FIG. 8 shows the flow of processing for estimating the shape by photographing the finger of the person to be detected.
As a finger shape estimation process, image data of a finger photographed by the video camera 32 is captured (step S121), and the coordinate axis of the color space of the image data is set with the conversion characteristics set in the initial adjustment process described above. A personal color system conversion process for conversion is performed (step S122). A finger image is detected based on the image data obtained by converting the coordinate axis of the color space (step S123), and the position of the nail is detected (step S124).
When each data is detected, the data having the highest similarity is searched from the database based on the detected data, and the angle data of all joints corresponding to the searched data is read (step S125). When the data of each joint is read, a command for setting the joint actuator 52 of the robot hand 50 to an angle corresponding to the read joint angle is generated (step S126). When the process so far is completed, the process returns to step S121, and the process is repeated from the reading of the image. The shape indicated by the angle of each joint read in step S125 corresponds to the estimated finger shape.

Next, details of each process will be described with reference to FIG.
First, the details of the process of extracting the image feature amount performed by the image feature amount extraction means 12 when the database is constructed will be described with reference to the flowchart of FIG. Here, noise removal processing is performed on the image data (step S11), and the image data from which the noise has been removed is binarized (step S12). From the binarized image data, the outline of the hand region is detected (step S13). In order to perform this contour line detection process, higher-order local autocorrelation patterns are calculated for all the pixels indicated by the image data (step S14).
FIG. 10 shows an example of a local pattern for detecting higher-order local autocorrelation features. In this example, the pixel arrangement is shown in block units of 9 pixels of 3 vertical pixels × 3 horizontal pixels, classified into 25 types of patterns from pattern M1 to pattern M25, and the outline is shown in black. In this case, it is calculated that the pattern has a corresponding number, and the contour line of the finger is indicated by a combination of the patterns.
The finger contour data thus obtained is registered in the database.
Note that it is not always necessary to use high-order local autocorrelation features for image feature extraction, and it is sufficient that features included in an image can be extracted by lower-order features.
In the flowchart of FIG. 9, the contour detection processing is performed as the finger shape detection processing from the image. However, for example, the finger region itself in the image may be detected. For example, a finger color portion (data glove color portion) in the image may be detected to detect not the contour of the finger but the region where the finger is present.

Next, details of the nail position calculation processing performed by the nail position calculation means 13 at the time of database construction will be described with reference to the flowchart of FIG. Here, noise removal processing is performed on the image data (step S21), and the noise-removed image data is divided into a black portion (background portion of the image), a white portion (data glove main body portion of the image), and a red portion. It is ternarized into a part (a part of the nail of the data glove) (step S22). This color coding is an example when photographing is performed on a black background.
Then, a red portion is detected as a nail region in the ternary image data (step S23). Further, the center-of-gravity position of each detected nail region is calculated (step S24). The calculated data on the center of gravity of each nail is registered in the database as nail position data.
FIG. 13 is a diagram showing an image of the nail position calculation process. Here, it is assumed that an image as shown in FIG. 13A is obtained as an image obtained by photographing a finger. In the process of extracting the feature amount of the image, the contour of the finger is detected, but in the nail position calculation process, only the nail area of five fingers is extracted as shown in FIG. . And as shown in FIG.13 (c), the gravity center position ((x1, y1) etc.) of each area | region is calculated for every area | region of each nail | claw. The calculated barycentric position is registered in the database.
In this example, the position data of each nail is the data of the center of gravity, but it may be data other than the position of the center of gravity. For example, a simple center may be used. Alternatively, the data of each nail region (that is, the data of the nail region detected in step S23 in the flowchart of FIG. 11) may be directly registered in the database.

Next, a process for creating a personal color system, which is performed as an initial adjustment when estimating the shape of a finger, will be described with reference to the flowchart of FIG.
Before describing the flowchart of FIG. 12, the reason for performing the personal color system creation process will be described.
This personal color system creation process is performed by photographing a finger whose shape is to be estimated and based on the skin color distribution of the finger. That is, FIG. 18A shows the color components of the image data in which two color signal components (color difference signal components) are orthogonally crossed. In FIG. 18A, the color components are represented by general color space coordinates indicated by the X axis and the Y axis, which are the color difference signal spaces. The human skin color region F1 such as a finger is a region which is shown in an elongated and oval shape as shown in FIG. The nail color area NA is shown as a part of the skin color area F1. The nail color area NA is a slightly pale white area in the skin color. When estimating the shape of the finger, a process of detecting the nail color area NA is performed.

  In order to detect the skin color area F1 in the color space indicated by the X axis and the Y axis shown in FIG. 18A, for example, the range x1 to x2 of the skin color area F1 on the X axis and the Y axis Detection is possible by setting the range y1 to y2 of the skin color region F1. However, when such a range is set, as shown in FIG. 18A, a range including many areas other than the skin color area F1 is included. The part described as noise in FIG. 18A is an extra area.

Here, in this example, as shown in FIG. 18B, the coordinate axis Sa of the angle passing through the barycentric position of the skin color region and the coordinate axis Sb orthogonal to the axis Sa are set as the color space coordinates. A skin color region F1 is defined on the color space indicated by the axes Sa and Sb. In the example of FIG. 18B, a substantially oval skin color region F1 is defined by the range a1 to a2 on the axis Sa and the range b1 to b2 on the axis Sb. By defining in this way, it is possible to extract the skin color area F1 with almost no extra area.
As shown in FIG. 18B, the image data in the color space indicated by the X-axis and the Y-axis is converted into image data in the color space indicated by the Sa-axis and the Sb-axis with the axis angle shifted. This is a conversion process in the personal color system conversion means 21 in FIG.
Expressions for conversion to the color system are shown in the following [Expression 1] and [Expression 2].

  In the equation, R, G, and B are values of the RGB color system, and Y, I, and Q are values of the YIQ color system. The YIQ color system is a color system that is used in North American television broadcasting and is excellent in skin color expression.

By the way, there is an individual difference in the coordinate position on the color space of the skin color area F1 shown in FIG. That is, there are individual differences in the tone of the skin color, which is the color of the skin, and for example, as shown in FIG. 19, there is a slight difference between the skin color area F1 of one person and the skin color area F2 of another person. Therefore, in this example, the skin color region of the finger of the detection target person is detected by reading the reference image for personal color system conversion described in the flowchart of FIG. 7, and passes through the center of gravity of the detected skin color region. The individual difference of the conversion characteristics is adjusted so that the Sa axis to be set and the Sa axis orthogonal to the Sa axis are set.
The center of gravity of the skin color area is calculated by the following [Equation 3]. However, m is the number of appearances of the color, M is the number of pixels in the frame, and X is a point on the color difference space where the value of the Q axis is q and the value of the I axis is i.

  When the unit vector of the inclination obtained by the equation [3] is α (Q ′, I ′), the skin color axis Sa can be obtained by the following equation from the equation [2].

Similarly, the flesh color orthogonal axis Sb can also be calculated from the unit vector orthogonal to the unit vector α.
Note that the coordinate position of the skin color region changes depending on the lighting conditions at the time of shooting in addition to individual differences.

  The processing described so far is executed in the personal color system creation processing shown in the flowchart of FIG. That is, as shown in FIG. 12, first, a skin color distribution chart of a person is created from an image obtained by photographing a skin color part such as the hand of the person to be detected (step S31). When the skin color distribution chart is created and the skin color area is detected, a process for obtaining the center of gravity of the person's skin color area (skin color distribution) is performed (step S32). Then, an axis (Sa axis) that passes through the skin color region with the center of gravity position as an inclination is obtained (step S33). Further, an Sb axis orthogonal to the Sa axis is obtained (step S34).

Next, a specific processing example of image feature amount extraction when estimating the shape of the finger of the person to be detected will be described with reference to the flowchart of FIG.
This image feature quantity extraction processing is basically the same as the image feature quantity extraction processing at the time of database creation shown in the flowchart of FIG. However, the processing configuration is to detect from image data converted into the personal color system. That is, first, noise removal processing is performed on the image data (step S41), and the image data from which the noise has been removed is converted into data on the coordinate system of the personal color system to extract a finger region (ie, a skin color region) ( Step S42). A contour line of the hand region is detected from the extracted finger region (step S43). In order to perform this contour line detection process, higher-order local autocorrelation patterns are calculated for all the pixels indicated by the image data (step S44). As a local pattern for detecting higher-order local autocorrelation features, for example, the pattern shown in FIG. 10 can be applied.

Next, a specific processing example of nail position calculation at the time of estimating the finger shape of the person to be detected will be described with reference to the flowchart of FIG.
This nail position calculation process is a process for detecting the nail area NA in the skin color area F1 described with reference to FIG. First, noise removal processing of image data is performed (step S51), and the image data from which the noise has been removed is converted into coordinate data of a personal color system to extract a finger region (ie, a skin color region) (step S51). S52). Further, by referring only to the value of the axis (Y axis) orthogonal to the skin color axis (X axis), the nail area NA is further extracted from the skin color area (step S53). Then, the barycentric position of the coordinates of the extracted image of the nail region is calculated (step S54). This barycentric position is data used as position data for each nail.

FIG. 16 shows a specific processing example for extracting a finger region (that is, a skin color region) from each pixel in the image data.
First, the pixel number i is set to 0 (step S61), and the value (a, b) indicated by the personal color system of the pixel number is calculated (step S62). Values (a, b) are values of the Sa axis and Sb axis. It is determined whether or not the obtained values (a, b) are values within the ranges (a1 to a2, b1 to b2) shown in FIG. 16B (step S63). If it is determined by this determination that the pixel is within the corresponding range, it is determined that the current pixel i is a finger region (step S64). If it is determined that it is not within the corresponding range, it is determined that the current pixel i is not a finger region (step S65).

  After the determination in steps S64 and S65, one pixel number i is added (step S66), and it is determined whether or not the added pixel number i is larger than the last pixel number (step S67). If the value is not larger than the last pixel number, the process returns to step S62. In this way, the skin color region that is the finger region is extracted from the image data.

FIG. 17 shows a specific processing example for extracting the nail region from each pixel in the image data.
First, the pixel number i is set to 0 (step S71), and the value (a, b) indicated by the personal color system of the pixel number is calculated (step S72). It is determined whether or not the value Sb indicated by the axis B in the obtained values (a, b) is a value corresponding to the nail range NA (step S73). Here, the value corresponding to the nail range NA is, for example, determination of whether the value is within the range from the value b2 to the value b3 shown in FIG. If it is determined by this determination that the pixel is within the corresponding range, it is determined that the current pixel i is a nail region (step S74). If it is determined that it is not within the corresponding range, it is determined that the current pixel i is not a nail region (step S75).

After the determination in steps S74 and S75, one pixel number i is added (step S76), and it is determined whether or not the added pixel number i is larger than the last pixel number (step S77). If the value is not larger than the last pixel number, the process returns to step S72. In this way, the nail region is extracted from the image data.
Note that the processing for extracting the finger region in FIG. 16 and the processing for extracting the nail region in FIG. 17 may be performed simultaneously as one processing.
By extracting the nail region in this manner, the nail can be accurately detected from the captured image.

As described above, according to the present embodiment, the contour or range of the finger is detected from the image, the position of the nail in the finger indicated by the contour or the like is detected, and the shape of the finger and the position of the nail are determined. Since the shape of the finger is estimated based on the above, the shape of the finger can be estimated very accurately. That is, in the case of only the contour of the finger, there is a possibility that the bending state of the finger joint and the bending state of the wrist may be erroneously estimated. The estimation can be reduced as much as possible.
Further, in the case of the present embodiment, the finger detection and the nail detection are performed in the color space converted into the personal color system, so as described in FIGS. 16 (a) and 16 (b), There are few noise components in the extracted skin color area, and good detection can be performed. Moreover, since the area to be detected is much smaller than the area indicated by the normal color space indicated by the X axis and the Y axis, the flowcharts of FIGS. 16 and 17 for extracting the finger area and the nail area are shown. The processing shown can be performed with a small amount of calculation processing, and each extraction processing can be performed at high speed.

  In the above-described embodiment, in order to detect the characteristics of the personal color system, initial adjustment for calculating the conversion characteristics of the personal color system is performed from an image obtained by actually photographing the skin. With this processing configuration, the conversion characteristics of the personal color system may be set. For example, the person to be detected may make a selection according to his / her skin color before performing the operation. Specifically, as an initial setting operation, each skin color can be selected from multiple types such as “Is the skin color white, black, or normal” by operating buttons, etc. The conversion characteristics of the personal color system may be set according to the above. In addition, the conversion characteristics of the personal color system differ depending on the race, such as yellow, white, and black, so the conversion characteristics of the personal color system should be set in the operation for setting such a race. May be.

  In the embodiment described above, as a process for detecting the position of the nail when the database is constructed, a mark corresponding to the nail is attached to the position where the fingernail is located on the data glove in red or the like, and the mark of the nail is marked. The nail position is detected and detected, but the nail position may be detected by another process. For example, after data indicating the shape of a finger is taken into a database, a three-dimensional image of the finger is generated and displayed by computer graphics based on the data on the database. Then, while the operator confirms the image, the process of attaching the nail to the finger in the three-dimensional image is performed by the operator, and the nail position set by the operation is used as the nail position data. May be.

  In the above-described embodiment, the processing configuration is such that the joints of the robot hand are controlled to correspond to each other based on the shape of the fingers estimated from the image. It may be performed based on the shape estimation.

  In the above-described embodiment, the database device 10 and the finger shape estimation device 20 are described as examples configured as dedicated devices. However, for example, a personal computer device that performs various data processing, and a necessary peripheral device such as a camera. It is also possible to implement the processing of the present invention by mounting software (program) for creating a database according to the present invention and performing finger shape discrimination (estimation) processing while referring to the database. In this case, the program for executing the processing of the present invention may be configured to be downloaded via a transmission means such as the Internet, in addition to being distributed on a medium such as an optical disk or a semiconductor memory.

It is a block diagram which shows the system configuration example by one embodiment of this invention. It is explanatory drawing which shows the example of the data glove utilized for construction of the database by one embodiment of this invention. It is a perspective view which shows the example of the robot hand controlled by one embodiment of this invention. It is a perspective view which shows the example which controls the image of HMD by one embodiment of this invention. It is explanatory drawing which shows the example of the finger shape (a) estimated by HDM of FIG. 4, and the image displayed based on the estimated finger shape. It is a flowchart which shows the flow of the database construction process by one embodiment of this invention. It is a flowchart which shows the example of an initial setting process by one embodiment of this invention. It is a flowchart which shows the flow of the finger discrimination process by one embodiment of this invention. It is a flowchart which shows the image feature-value extraction process example at the time of the database construction by one embodiment of this invention. It is explanatory drawing which shows the example of the local pattern for the high-order local autocorrelation characteristic by one embodiment of this invention. It is a flowchart which shows the nail | claw position calculation process example at the time of the database construction by one embodiment of this invention. It is a flowchart which shows the creation processing example of the personal color system by one embodiment of this invention. It is explanatory drawing which showed the example which extracted the gravity center position of the nail | claw using the personal color system by one embodiment of this invention. It is a flowchart which shows the image feature-value extraction process example at the time of the finger shape estimation by one embodiment of this invention. It is a flowchart which shows the nail | claw position calculation process example at the time of the finger shape estimation by one embodiment of this invention. It is a flowchart which shows the example of a process which extracts a finger area | region from the image by one embodiment of this invention. It is a flowchart which shows the process example which extracts a nail | claw area | region from the image by one embodiment of this invention. It is a characteristic view which shows the example of the skin color area | region and nail | claw area | region by one embodiment of this invention. It is the characteristic view which showed the personal color system by one embodiment of this invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 ... Database apparatus, 11 ... Image data storage means, 12 ... Image feature-value extraction means, 13 ... Nail position calculation means, 14 ... Image and nail position corresponding | compatible operation command storage means, 15 ... Sampling means, 16 ... Time series joint angle Data storage means, 20 ... finger shape storage means, 21 ... personal color system conversion means, 23 ... image data storage means, 24 ... image feature quantity extraction means, 25 ... nail position calculation means, 26 ... data specification and operation command generation Means 27: Driving means 31, 32 ... Video camera, 40 ... Data glove, 41 ... Angle sensor, 50 ... Robot hand, 51 ... Supporting part, 52 ... Actuator for joint, 90 ... HMD with camera, 91 ... 3-axis Position sensor, 92 ... 3D virtual object, 93R ... Right hand with 3D CG, 93L ... Left hand with 3D CG

Claims (10)

A database for storing image data corresponding to various shapes of fingers and nail position data indicating the position of the nail when the shape is indicated by the image data;
Nail detection means for detecting the position of the nail from image data obtained by photographing the detection target finger;
Collation means for collating the position of the nail detected by the nail detection means with the nail position data stored in the database;
Based on the nail position data in which the match is detected by the collating means, the image data stored in the database corresponding to the nail position data is read, and the finger shape indicated by the image data is determined as the detection target. An apparatus for estimating a finger shape, comprising: estimation means for estimating the shape of a finger. The finger shape estimation apparatus according to claim 1,
The database is a finger shape estimation device for storing finger contour information together and collating both nail position data and finger contour information stored in the database. The finger shape estimation apparatus according to claim 1,
The finger shape estimation apparatus, wherein the database stores finger shape data corresponding to image data. In the finger shape estimation apparatus according to claim 3,
The finger shape data stores finger joint angle data, and from the joint angle data stored in the database corresponding to the nail position data collated by the collating unit, the joint angle of the finger by the estimating unit The finger shape estimation apparatus characterized by estimating. In the finger shape estimation apparatus according to claim 3,
The shape data of the finger stores three-dimensional coordinate data, and the three-dimensional coordinate of the finger is estimated by the estimation unit from the three-dimensional coordinate data stored in the database corresponding to the nail position data verified by the verification unit. A finger shape estimation apparatus characterized by estimating data. The finger shape estimation apparatus according to claim 1,
The nail detection means extracts the skin color portion of the finger and the nail color portion included in the skin color portion from the image data obtained by photographing the finger to be detected, and extracts the nail color of the extracted nail color. A finger shape estimation device characterized by detecting a position of a nail from a portion. The finger shape estimation apparatus according to claim 6,
The nail detection means extracts a nail region from a color space expressed by a first axis passing near the color distribution region of a finger to be detected and a second axis orthogonal to the first axis. A finger shape estimation device. The finger shape estimation apparatus according to claim 7,
The finger shape estimation apparatus according to claim 1, wherein the angular position of each axis of the color space set by the first and second axes is set based on image data obtained by photographing a skin color portion of the detection subject in advance. Database configuration processing for storing image data corresponding to various shapes of fingers and nail position data indicating the position of the nail when the shape is indicated by the image data, and configuring a database;
Nail position detection processing for detecting the position of the nail from image data obtained by photographing the finger of the detection target;
A collation process for collating the nail position detected in the nail detection process with the nail position data stored in the database;
Based on the nail position data for which a match is detected in the matching process, the image data stored in the database corresponding to the nail position data is read, and the finger shape indicated by the image data is determined as the detection target. A finger shape estimation method comprising: determination processing for determining a finger shape. In a program that is mounted on a computer device and executes processing,
Database configuration processing for storing image data corresponding to various shapes of fingers and nail position data indicating the position of the nail when the shape is indicated by the image data, and configuring a database;
Nail position detection processing for detecting the position of the nail from image data obtained by photographing the finger of the detection target;
A collation process for collating the position of the nail detected in the nail detection process with the nail position data stored in the database;
Based on the nail position data for which a match is detected in the matching process, the image data stored in the database corresponding to the nail position data is read, and the finger shape indicated by the image data is determined as the detection target. A program for executing estimation processing for estimating the shape of a finger.

Images