JP6066093B2 - Finger shape estimation device, finger shape estimation method, and finger shape estimation program - Google Patents

Description

  The present invention relates to a finger shape estimation device, a finger shape estimation method, and a finger shape estimation program, and in particular, a finger shape estimation device and a finger shape estimation method for estimating the shape and movement of a finger in an image at high speed and with high accuracy. And a finger shape estimation program.

  Conventionally, for example, if an object has a fixed shape, it is possible to cause the robot or the like to grip the object by automatic control. However, it is difficult to freely grip objects of various shapes as humans think.

  Therefore, conventionally, there is a mechanism in which a sensor or marker is attached to a human finger, the movement of the human finger is detected from the attached sensor or marker, and the robot is driven according to the detected finger movement. If the robot or the like can be gesture-driven without using the above-described sensor or marker, the robot can be moved freely by remote operation in a state where it is intuitive and has a high degree of freedom and there are few restrictions during input. You can make the robot do the work. In addition, since an information communication terminal or the like can be operated by behaving in the same way as daily operations, prior learning of the operation method is not necessary.

  Conventionally, for example, a technique for performing a similarity search using a database and determining a finger shape using an image feature amount obtained from an outline of a finger read from an image, that is, a finger motion capture is used. (See, for example, Patent Documents 1 and 2).

International Publication No. 2005/046942 Pamphlet International Publication No. 2009/147904

  However, in the technique as disclosed in Patent Document 1, an image obtained by extracting only the contour line of a finger read from each of an input image and a database verification image is divided into 64, and a vertical line, horizontal line, diagonal line, The finger shape is expressed by image feature amounts corresponding to broken lines, dots, and the like. Therefore, if a person whose finger is significantly thicker or longer than the finger shape registered in the database uses the system, even if the feature amount of the same finger shape is calculated, it becomes a different value. In a registered database, information that should fit in one classification is stored in two different classifications, which may cause erroneous estimation.

  Further, in the technique as disclosed in Patent Document 2, each image of the captured image and the image for verification is divided into 64 and each divided region is expressed by a 25-dimensional image feature amount. It has dimension features. For this reason, in the technique as disclosed in Patent Document 2, there is a limit to the types of finger shapes that can be uploaded to the memory of a computer, and in order to perform real-time processing at a video rate or twice as fast as that, at most. Only about 30,000 types of shapes can be estimated. As a result, it is difficult for the technique disclosed in Patent Document 2 to estimate the finger shape having individual differences at high speed and with high accuracy.

  In other words, in the conventional method, as a means for dealing with individual differences, data (data sets) corresponding to individual differences must be newly added to the database, which is a great effort and increases the time required for collation processing by the database. End up. Here, individual differences include, for example, differences in hand shape (bone length, thickness, palm-to-finger ratio), hand movement (joint range of motion, and how to take posture within it), etc. including.

  The present invention has been made in view of the above-described problems, and is a finger shape estimation device, a finger shape estimation method, and a finger shape estimation program for estimating the shape and movement of a finger in an image at high speed and with high accuracy. The purpose is to provide.

  In order to solve the above-described problems, the present invention has the following features.

The finger shape estimation apparatus of the present invention corresponds to the ridge line shape of a finger included in the image by analyzing an image acquired by the image acquisition unit that acquires an image including the finger shape and the image acquisition unit. An image analysis unit that acquires a first feature amount, and a second feature amount corresponding to a predetermined finger shape set in advance is accumulated based on the first feature amount obtained by the image analysis unit. And a finger shape estimation unit that estimates a finger shape corresponding to the first feature amount, the image analysis unit uses the finger image included in the image as a foreground, With the image other than the finger image as a background, the distance from the background image in the foreground image is regarded as a height so that the image is regarded as one mountain-shaped image, and the ridge line shape information is obtained from the mountain-shaped image. To get .

In addition, the finger shape estimation method of the present invention obtains an image including a finger shape, analyzes the acquired image, and corresponds to a ridge line shape of a finger included in the image. Acquiring a quantity, and referring to a database for collation in which a second feature quantity corresponding to a predetermined finger shape set in advance is stored based on the first feature quantity, and the first feature look including the estimating a human hand posture corresponding to the amount, the first obtaining a feature amount, and the hand image included in the image and the foreground, the background of images other than該手finger images, wherein Considering the image as one mountain-shaped image by regarding the distance from the background image in the foreground image as a height, obtaining the ridge line shape information from the mountain-shaped image .

  Moreover, the finger shape estimation program of the present invention is a finger shape estimation program for causing each processing of the finger shape estimation method of the present invention to be implemented in an information processing apparatus and executed.

  According to the present invention, the shape and movement of a finger in an image can be estimated at high speed and with high accuracy.

FIG. 1 is a diagram illustrating an example of a functional configuration of the finger shape estimation apparatus according to the first embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration capable of realizing the finger shape estimation process according to the first embodiment. FIG. 3 is a flowchart illustrating an example of a finger shape estimation process procedure according to the first embodiment. FIG. 4 is a diagram for explaining the outline of the collation processing in the first embodiment. FIG. 5 is a flowchart illustrating an example of the procedure of the collation database construction process in the first embodiment. 6A to 6D are diagrams illustrating examples of a plurality of finger shape images obtained using a data glove. 7A and 7B are diagrams illustrating an example of a database structure. FIG. 8 is a diagram illustrating an example of various parameters necessary for calculating the image shape ratio. 9A to 9C are diagrams illustrating an example of data serving as a reference for acquiring image feature amounts. 10A and 10B are diagrams illustrating examples of ridge line information extraction results. FIG. 11 is a diagram for explaining high-order local autocorrelation processing performed on the ridge line information of each divided region of the 8 × 8 divided ridge line image. 12A to 12C are diagrams illustrating an example of a pixel moving method. FIG. 13 is a flowchart illustrating an example of a processing procedure for acquiring a ridge line vector from contour scanning in the first embodiment. FIG. 14 is a flowchart illustrating an example of the processing procedure of the second stage of the collation processing. FIGS. 15A and 15B are diagrams illustrating an example of the difference between the contour line information and the ridge line information obtained when the finger shape is the same between the bare hand state and the military hand state. 16A to 16C are diagrams illustrating comparison results of joint angles. FIG. 17 is a diagram illustrating an example of an error average and an error standard deviation. 18A and 18B are diagrams showing an application example of the finger shape estimation apparatus of the present invention. FIG. 19 is a diagram illustrating an example of a functional configuration of the nail region extraction device according to the second embodiment. FIG. 20 is a diagram illustrating an example of a hardware configuration capable of realizing the nail region extraction process according to the second embodiment. FIG. 21 is a flowchart illustrating an example of a nail region extraction processing procedure according to the second embodiment. FIG. 22 is a diagram illustrating an example of a pixel distribution model in the RGB color space of a finger image. FIG. 23 is a flowchart illustrating a specific example of nail region extraction processing in the second embodiment. 24A to 24C are diagrams illustrating an example of coordinate conversion based on the principal component axis. 25A and 25B are diagrams illustrating an example of a separation plane position determination method. FIGS. 26A and 26B are diagrams showing nail region portions extracted in the second embodiment. FIG. 27 is a diagram illustrating an example of a skin distribution position with a high probability of erroneous extraction. 28A and 28B are diagrams illustrating examples of execution results when the nail region is re-extracted and when the erroneously extracted skin region is re-extracted, respectively. 29A to 29C are diagrams for describing the evaluation results in the second embodiment. FIG. 30 is a diagram illustrating an application example of the nail region extraction device according to the second embodiment.

1. 1st Embodiment <About the finger shape estimation technique of this invention>
In the present invention, when dealing with the above-mentioned problem of individual differences in fingers, the feature corresponding to the information of the contour line in order to enable estimation without necessarily adding individual information to the database. Instead of calculating the amount, for example, paying attention to information passing through the center of the finger, the feature amount corresponding to this information is calculated. As a result, the same feature amount can be acquired even when the thickness of the finger is greatly different or the length of the finger is somewhat different. Specifically, first, for example, a hand image (hand image) included in the image is set as the foreground image, and other images are set as the background image. Next, in each pixel of the foreground image (hand image), the distance (pixel) between the pixel and the pixel of the background pixel closest to the pixel is set to a height, and the hand image is simply referred to as a mountain image (hereinafter simply referred to as a mountain). ) Then, ridge line information (information passing through the center of the finger) that can be drawn with respect to the mountain is acquired, and the acquired ridge line information (ridge line shape) is used for finger shape estimation. Accordingly, in the present invention, it is not necessary to add a database corresponding to individual differences, so that the finger shape can be estimated at high speed and with high accuracy.

  Moreover, since this invention estimates using the information which passes along the center of a finger | toe, the feature-value corresponding to this information can be acquired by vectorization of this information. Specifically, for example, when acquiring a feature amount corresponding to the above-described ridge line information, the feature amount of the information is calculated by vectorization. Since the number of vectors obtained by vectorization of the ridge line information varies depending on the shape of the finger, the feature quantity dimension may not be stable, but it may be about 100 dimensions at most. Thereby, the feature quantity dimension of each data set of the database for collation can be significantly reduced. Conversely, even when a large number of data sets are increased, the same high speed as in the conventional method can be maintained. Furthermore, since the ridge line information is not easily influenced by individual differences, it is not necessary to add a new large-scale database for matching corresponding to individual differences later.

  That is, the present invention has the following features, for example, as compared with the conventional method. In addition, the following description is an example and the characteristic of this invention is not limited to this.

<Information used for finger shape estimation>
For example, in the prior art as shown in Patent Document 2, an input image is reduced to a vertical 64 [pixel] × horizontal 64 [pixel] image, and an image feature is obtained by acquiring a feature amount corresponding to the outline information. Expressed. However, in the present invention, as described above, for example, when a hand image is a foreground and other images are used as a background, the distance from the background image of the foreground image is regarded as a height, so that the hand image is regarded as one mountain. . Then, the ridge line that can be drawn on the mountain is used for estimating the finger shape. Since the ridge line information is information passing through the center of the finger, when the feature amount corresponding to the ridge line information is calculated, the feature amount does not change greatly. Accordingly, it is possible to perform finger shape estimation corresponding to individual differences without newly registering a database for dealing with the above-described problem of individual differences.

<Number of dimensions of image features>
For example, in the prior art as shown in Patent Document 2, one finger data set or input image is divided into a total of 64 sections of 8 × 8 in length, and the image features of each section are divided into higher-order local autocorrelations. It is expressed by low-order feature quantities such as 25 patterns of points, line segments, broken lines, and edges corresponding to functions. As a result, one finger image had a total of 1600 dimensions of 8 × 8 sections × 25 patterns.

  On the other hand, in the present invention, the feature amount of the ridge line information is obtained by vectorizing the extracted ridge line information. The number of feature quantity dimensions obtained by vectorization varies depending on the input image, but is about 100 dimensions at the maximum. Therefore, the number of feature dimensions is at least about 1/16 compared to the conventional 1600 dimensions, which can greatly reduce the image feature.

<Database scale>
For example, the database scale of the prior art as shown in Patent Document 2 is about 30000 data sets. The number is a result of a database constructed by a large number of preliminary experiments, in particular, so that each finger can estimate the full bending and full extension and the intermediate posture with high accuracy. The number of about 30000 sets is not necessarily the upper limit that can be uploaded to the memory (storage unit) of the computer, but if the resolution that can be estimated is made finer, the number of digits in the required data set will increase explosively, so The number was close to the upper limit.

  However, according to the present invention, the number of feature dimensions per data set is greatly reduced to about 1/16, so that the information (number of data sets) that can be uploaded to the memory of the computer is greatly increased. Since the database scale can be increased by 16 times, a database having about 480000 data sets can be created.

<Estimation resolution>
As described above, the database scale of the prior art was about 30000 data sets. Assuming that the three joints of the four fingers each have one degree of freedom (ie, move in a fixed ratio), the thumb has three degrees of freedom, and the four fingers open and close has one degree of freedom, the types of finger shapes are the four fingers and the thumb The combination of each of the four-step postures and the two-step postures of opening and closing the four fingers will exceed 30,000 types. Actually, the four-finger opening / closing (that is, four-finger inversion / extraction) has a great influence on the quality of the estimation, so it is necessary to increase the stage of the four-finger opening / closing posture. That is, in the prior art, the resolution of finger shape estimation is a rough resolution including complete bending and complete extension and about 1 to 3 kinds of intermediate postures. If a data set corresponding to individual differences is also prepared, the conventional database scale of 30000 data sets is not sufficient.

  On the other hand, in the present invention, as described above, the database scale can be increased to about 16 times the database scale even at the same processing speed. If the postures of the four fingers and the thumb 3 joints are each provided in five stages, and the four-finger opening / closing postures are provided in three stages, there are about 250,000 types of all combinations. In addition, since there is no need for a data set to deal with individual differences, the expanded data set can contain data for improving the resolution, giving a slightly finer resolution to the index finger, middle finger, and thumb. This makes it possible to construct a database having about 400,000 data sets.

  Hereinafter, embodiments in which a finger shape estimation device, a finger shape estimation method, and a finger shape estimation program according to the present invention are suitably implemented will be described with reference to the drawings.

<Finger shape estimation device: functional configuration example>
Next, a functional configuration example of the finger shape estimation apparatus according to the first embodiment will be described with reference to the drawings. FIG. 1 is a diagram illustrating an example of a functional configuration of the finger shape estimation apparatus according to the present embodiment. 1 includes an input unit 11, an output unit 12, a storage unit 13, an image acquisition unit 14, a database construction unit 15, an image analysis unit 16, a collation unit 17, and a finger. The shape estimation unit 18, the transmission / reception unit 19, and the control unit 20 are configured.

  The input unit 11 accepts input such as start / end of various instructions such as an image acquisition instruction, a database construction hand instruction, a database construction instruction, an image analysis instruction, a collation instruction, a finger shape estimation instruction, and a transmission / reception instruction from a user or the like. The input unit 11 is composed of a pointing device such as a keyboard or a mouse if it is a general-purpose computer such as a PC (Personal Computer), and each operation button group if it is an information terminal device or game device such as a smartphone or a mobile phone. Etc. The input unit 11 may have a voice input function for inputting voice such as the above-described instructions by voice or the like.

  The output unit 12 outputs information such as content input by the input unit 11 and content executed based on the input content. Specifically, the output unit 12 performs screen display, audio output, and the like of the acquired image, database construction results, image analysis results, verification results, processing results in each configuration such as finger shape estimation results, and the like. The output unit 12 includes a display, a speaker, a robot, and the like.

  Further, the output unit 12 may have a printing function such as a printer, and the above-described output contents can be printed on various printing media such as paper and provided to the user or the like.

  The accumulating unit 13 accumulates various information necessary in the present embodiment, various data at the time of execution of the process or after execution of the process, and the like. Specifically, the storage unit 13 includes images stored in advance, images obtained by shooting acquired by the image acquisition unit 14 (for example, including time-series images such as videos), and the like. accumulate. In addition, the storage unit 13 includes the contents of the database obtained by the database construction unit 15, the analysis result obtained by the image analysis unit 16, the collation result obtained by the collation unit 17, and the estimation obtained by the finger shape estimation unit 18. Accumulate results. Further, the storage unit 13 can read out various data stored as necessary.

  The image acquisition unit 14 acquires, for example, an image or video captured by the imaging device 21 or the like. For convenience of explanation, it is assumed that all images acquired by the image acquisition unit 14 include fingers, but the present invention is not limited to this.

  Here, in the present embodiment, the imaging device 21 is provided outside the finger shape estimation device 10. However, the present invention is not limited to this, and the imaging device 21 is built in the finger shape estimation device 10, for example. May be. Further, the image and video acquired by the image acquisition unit 14 are not limited to the actual finger image or video captured by the imaging device 21, and for example, a model finger or photograph, a poster, or the like was captured. It may be an image, an image generated by CG (Computer Graphics) editing software, or the like.

  The image acquisition unit 14 can also acquire images, videos, and the like stored in an external device connected to the communication network, a database, and the like via the transmission / reception unit 19. The image acquired by the image acquisition unit 14 can be stored in the storage unit 13 and can be read from the storage unit 13 as necessary.

  The database construction unit 15 acquires necessary information such as movements of each joint of the hand from the data glove, which is obtained by the user wearing a data glove with a sensor, a marker, etc., if necessary. Thus, a database for collation required for finger shape estimation in this embodiment is constructed. Alternatively, an image generated by designating a finger joint angle may be acquired by CG editing software or the like, and a database for collation required for finger shape estimation in this embodiment may be constructed.

  The database constructed by the database construction unit 15 accumulates at least angle data and image feature amounts for a predetermined finger shape. Then, for example, the three data of the angle data (for example, joint angle etc.), the image shape ratio, and the image feature amount described above as one set (data set) with respect to the input image is a forearm rotation angle, You may build a database. Note that the above-described three data can be obtained by analyzing the input image by the image analysis unit 16 or the like, for example. Further, the above-described data set is not limited to the above-described three data, and for example, it is sufficient that at least one of the above-described three data is included.

  That is, in the present embodiment, “image feature amount using ridge line” and “joint angle information” are associated and stored in the database. In other words, in the present embodiment, for example, when the user's finger or the like is photographed by the imaging device 21 such as a camera, the “image feature amount using the ridge line” of the user's finger and the “ridge line using the ridge line” are used. “Image feature value” is compared and “joint angle information” of a data set associated with the most similar image feature value is output as an estimation result. Accordingly, the database constructed in the present embodiment requires, for example, image feature amounts of ridge lines and angle data. Further, the forearm rotation angle and the image shape ratio described above are additional data used for narrowing down the data, for example, and may not be stored as a database, but these additional data are stored in the database. By doing so, it is possible to realize efficient and highly accurate narrowing down.

  Further, the database construction unit 15 acquires the database used in the present embodiment from the external device connected by the communication network via the transmission / reception unit 19 when the database is already constructed and accumulated in the accumulation unit 13 or the like. If so, the database need not be constructed.

  The image analysis unit 16 analyzes the image (including video) acquired by the image acquisition unit 14. Specifically, the image analysis unit 16 removes a non-finger region such as a background, an arm, or a body from the image, and in which part (position, region) the finger or the like from the feature amount for each pixel in the image It is analyzed how the position of the object is projected, how the object such as a finger moves in the video, and the like. That is, the image analysis unit 16 performs numerical processing of the feature amount of the captured image such as a finger. Specifically, the image analysis unit 16 acquires ridge line information using, for example, a contour shape (contour line) of a finger, and acquires a feature amount from the acquired ridge line shape. Further, the image analysis unit 16 acquires two data of the image shape ratio and the above-described image feature amount with respect to the input image. However, the image shape ratio is not necessarily acquired, and data other than the image shape ratio data described above may be included.

  In addition, as an example of acquiring a contour line, for example, based on information on a luminance difference between adjacent pixels, the finger part and the background part can be separated from the image, and the contour line of the finger part can be acquired. The present invention is not limited to this.

  Based on the analysis result obtained by the image analysis unit 16 from the input image, the collation unit 17 collates the input image with a preset collation database, and performs similarity determination. Specifically, the collation unit 17 uses, for example, at least the image feature amount of the two data (image shape ratio and image feature amount) described above to calculate the finger shape included in the database and the finger shape in the input image. Perform verification.

  The finger shape estimation unit 18 estimates the finger shape corresponding to the finger in the image based on the collation result obtained by the collation unit 17. A specific method for estimating the hand shape in the hand shape estimating unit 18 will be described later.

  Further, the transmission / reception unit 19 acquires a desired external image (for example, a photographed image or video) from an external device that can be connected using a communication network or the like, an execution program for realizing the finger shape estimation process in the present invention, and the like. It is an interface to do. Moreover, the transmission / reception part 19 can transmit the various information obtained in the finger shape estimation apparatus 10 to an external device.

  The control unit 20 controls the entire components of the finger shape estimation device 10. Specifically, the control unit 20 performs control in each process such as image acquisition, database construction, image analysis, image matching, and finger shape estimation based on, for example, an instruction from the input unit 11 by a user or the like. Do.

  The imaging device 21 includes, for example, a digital camera, a high-precision camera, and the like, and acquires images and videos of the user's actual fingers and model fingers. Note that only one imaging device 21 may be provided, or a plurality of imaging devices 21 may be provided so that fingers can be photographed simultaneously from different directions.

<Hand shape estimation device 10: hardware configuration>
Here, in the hand shape estimation apparatus 10 described above, an execution program (hand shape estimation program) or the like as software capable of causing a computer (information processing apparatus, hardware) to execute each function is generated. By installing the execution program on an information terminal device such as a general-purpose personal computer, a server, an information terminal device such as a smartphone or a mobile phone, or a game machine, the finger shape estimation process or the like in the present invention can be realized.

  Here, a hardware configuration example of a computer capable of realizing the finger shape estimation process in the present embodiment will be described with reference to the drawings. FIG. 2 is a diagram illustrating an example of a hardware configuration capable of realizing the finger shape estimation process according to the present embodiment.

  2 includes an input device 31, an output device 32, a drive device 33, an auxiliary storage device 34, a memory device 35, a CPU (Central Processing Unit) 36 that performs various controls, and a network connection device. 37, and these are connected to each other by a system bus B.

  The input device 31 has a pointing device such as a keyboard and a mouse operated by a user or the like, and inputs various operation signals such as execution of a program from the user or the like. The input device 31 may include an image input unit that inputs an image captured from the imaging device 21 such as a camera.

  The output device 32 has a display for displaying various windows and data necessary for operating the computer main body for performing the processing according to the present invention, and displays the program execution progress and results by the control program of the CPU 36. can do.

  Here, the execution program installed in the computer main body in the present invention is provided by, for example, a portable recording medium 38 such as a USB (Universal Serial Bus) memory or a CD-ROM. The recording medium 38 on which the program is recorded can be set in the drive device 33, and the execution program included in the recording medium 38 is installed in the auxiliary storage device 34 from the recording medium 38 via the drive device 33.

  The auxiliary storage device 34 is a storage device such as a hard disk, and can store an execution program according to the present invention, a control program provided in a computer, and the like, and can input and output them as necessary.

  The memory device 35 stores an execution program read from the auxiliary storage device 34 by the CPU 36. The memory device 35 includes a ROM (Read Only Memory), a RAM (Random Access Memory), and the like.

  The CPU 36 controls processing of the entire computer, such as various operations and input / output of data with each hardware component, based on a control program such as an OS (Operating System) and an execution program stored in the memory device 35. Thus, each process in the finger shape estimation process can be realized. Various information necessary during the execution of the program can be acquired from the auxiliary storage device 34, and the execution result and the like can also be stored in the auxiliary storage device 34.

  The network connection device 37 acquires an execution program from another terminal connected to the communication network by connecting to a communication network or the like, or an execution result obtained by executing the program or an execution in the present invention The program itself can be provided to other terminals.

  With the hardware configuration as described above, the finger shape estimation process in the present invention can be executed. In addition, by installing the program, the finger shape estimation process according to the present invention can be easily realized by a general-purpose personal computer or the like.

  Next, the finger shape estimation process in the above-described finger shape estimation program will be specifically described.

<Finger shape estimation processing procedure>
First, the finger shape estimation processing procedure in the present embodiment will be described. FIG. 3 is a flowchart showing an example of a finger shape estimation processing procedure in the present embodiment. The operation of each unit in various processes described below is controlled by the control unit 20 (CPU 36).

  In the finger shape estimation process shown in FIG. 3, first, the control unit 20 determines whether or not there is a collation database for estimating the finger shape (S01). Specifically, the control unit 20 determines whether or not the collation database is stored in the storage unit 13 or the like in advance, or is acquired from an external device or the like through a communication network or the like. Here, when there is no collation database (NO in the process of S01), the control unit 20 controls each unit to construct a collation database (S02).

  However, in the present embodiment, the processes of S01 and S02 are not essential. For example, in the case of NO in the process of S01, the process may be terminated.

  Further, when there is a collation database in the process of S01 (YES in S01), or when a collation database is constructed in the process of S02, the image acquisition unit 14 selects a finger whose hand shape needs to be estimated. The image including it is acquired (S03). Next, the image analysis unit 16 analyzes the acquired image (S04). As the analysis processing in S04, for example, processing such as obtaining an image shape ratio or calculating an image feature amount (first feature amount) from a ridge line shape of a finger is performed. It is not limited.

  Further, in the finger shape estimation process, the collation unit 17 uses the collation database obtained by the process of S02 or the like based on the analysis result obtained in the process of S04 or the collation database previously stored in the storage unit 13 or the like. Are compared with the image acquired in S03 (S05). Next, the finger shape estimation unit 18 estimates the finger shape in the acquired image (S06), and associates the finger shape with the image feature ratio, the image feature amount of the ridge line shape, and the like stored in the matching database. The angle data is output as an estimation result (S07).

  Next, the control unit 20 determines whether or not to end the process (S08). If the process does not end (NO in S08), the control unit 20 returns to S03, and the control unit 20 performs, for example, continuous images, that is, videos. The above processing is performed and the results are output in time series, or the above processing is performed on another image.

  Moreover, in the process of S08, when a process is complete | finished by a user's instruction | indication etc. (in S08, YES), the control part 20 complete | finishes a finger shape estimation process.

  As described above, the processes of S01 and S02 may not be included in the finger shape estimation process. That is, the database construction is performed asynchronously with the finger shape estimation process. Therefore, for example, only at the first estimation, the presence / absence of the database may be determined and the database may be constructed as shown in S01 and S02 described above. When the desired database is not found, there is an instruction from the user or the like. In some cases, a database may be constructed.

  Here, FIG. 4 is a diagram for explaining the outline of the collation processing (S05) in the present embodiment. As shown in FIG. 4, the collation processing according to the present embodiment can be roughly classified from the data sets included in the collation database, for example, by narrowing down the data set by limiting the search range by forearm rotation, image shape ratio, or the like. And so on (ie, narrowing down the data set by a predetermined shape parameter in the finger shape) and the second step of outputting the most similar data set by performing similarity calculation based on the image feature amount Process. In the present embodiment, for example, the data set of all databases and the acquired image are collated using a data set such as the image shape ratio (the number of dimensions is 3) of the input image described above.

  In other words, in the present embodiment, for example, the data set (data set group) remaining after the rough narrowing by the image shape ratio or the like is further subjected to precise similarity matching by the image feature amount (for example, about 100 dimensions). A similar data set is output as a finger shape estimation result. Thereby, the shape and movement of fingers in the acquired image can be estimated at high speed and with high accuracy.

<Database construction process>
Next, the procedure of the collation database construction process (S02) in this embodiment described above will be described using a flowchart. FIG. 5 is a flowchart showing an example of a verification database construction processing procedure in the present embodiment. The operation of each unit in various processes described below is controlled by the control unit 20 (CPU 36).

  In constructing the collation database, first, the image acquisition unit 14 acquires a plurality of finger-shaped images obtained using a data glove or the like (S11). Here, FIGS. 6A to 6D are diagrams illustrating an example of a plurality of finger shape images obtained using a data glove. When a user constructs a collation database for estimating a finger shape, the user acquires images based on preset finger shapes as shown in FIGS. The finger shape is not limited to the examples of FIGS. 6A to 6D, and many shapes are used.

  Moreover, in the process of S11, the image acquisition part 14 acquires the shape from at least 1 angle, such as an upper surface, a side surface, and diagonally forward, as each finger shape. In the present embodiment, an image from a predetermined direction with respect to one finger is handled as a data set.

  Next, the image analysis unit 16 acquires the joint angles of fingers included in each image (S12). The joint angle indicates, for example, finger joint angle data, but is not limited to this in the present invention, and may include, for example, a joint angle such as a forearm rotation angle. An example of acquiring the joint angle will be described later.

  In this embodiment, instead of acquiring a plurality of finger-shaped images and joint angle information using a data glove or the like, a finger image is generated using, for example, CG editing software, and the plurality of joint angle information and A plurality of corresponding finger-shaped images may be acquired.

  Next, the image analysis unit 16 calculates the image shape ratio of the fingers included in each image (S13). A method for calculating the image shape ratio in the process of S13 will be described later. In the present invention, it is not always necessary to calculate the image shape ratio. Next, the image analysis unit 16 acquires a ridge line shape based on the contours of fingers included in each image (S14). Specifically, in the process of S14, the ridge line is set by regarding the hand image as a mountain based on the image feature amount.

  Next, the image analysis unit 16 acquires vector information as an image feature amount from the acquired ridge line (S15). Then, the database construction unit 15 constructs a collation database composed of a set (data set) including the acquired joint angle (angle data), image shape ratio, and image feature amount (second feature amount) (S16).

  In the process of S16, it is only necessary to include the image feature amount described above and the joint angle (angle data) used for outputting the estimation result, and data other than the image shape ratio data described above is included. It may be. Here, the structure of the database described above will be specifically described.

<Database structure>
7A and 7B are diagrams illustrating an example of a database structure. In the present embodiment, for example, as shown in FIG. 7A, for example, angle data (JOINT ANGLES WITH WRISTRATION) including a finger joint angle and a forearm rotation angle in a predetermined finger shape, an image shape ratio (IMAGE ASPECTS), and Three data of image feature values (IMAGE FEATURES) are set as one data set (DATA SET), and a set of data sets for various finger shapes is set as a database (DATABASE). As for the angle data, for example, as shown in FIG. 7B, a gripping operation for the angle data may be set in advance using identification information or the like. Thereby, it is possible to efficiently acquire a gripping operation (movement of fingers) based on the angle data at the time of collation.

  In other words, when trying to control a remote robot by a user's (operator's) finger movement, for example, it is possible to quickly grasp a finger image captured by a camera rather than performing detailed finger shape estimation. If one of them is identified, a stable and high-speed remote robot operation can be realized. Therefore, in the present embodiment, as shown in FIG. 7B, at the time of database construction, for example, finger shapes that need to be identified and their individual difference data are intensively generated, and instead of finger joint angle data, Numbers (1, 2, 3, ...) are assigned.

  Thereby, in this embodiment, the most similar finger shape searched by the rough narrowing of the 1st step in the collation process mentioned above and the fine similarity collation of the 2nd step can be made into an estimation result. . In this case, the content output as the estimation result may not be the joint angle. For example, a gripping motion pattern number set in advance corresponding to the estimation result may be output.

<Finger joint angle>
Here, the finger joint angle included in the verification database is acquired by, for example, a data glove (for example, CyberGlove (registered trademark) manufactured by Virtual Technologies).

<Image shape ratio>
Next, a method for calculating the image shape ratio will be specifically described. FIG. 8 is a diagram illustrating an example of various parameters necessary for calculating the image shape ratio. In the method of calculating the image shape ratio, first, the finger area and the background are separated, and a labeling process is performed on the finger area. At that time, the pixel having the largest label number is set as a reference point, and the finger range is determined based on the reference point. Here, for example, the lower part of the finger range by the pixel corresponding to the label number of the reference point is set as the lower end of the finger range, and the upper end, right end, and left end of the finger range are determined so that the finger region just enters the finger range. The image shape ratio is represented by three values of verticalness, upperness and rightness, and is defined by the following equations (1) to (3), respectively.

Here, in the above-mentioned _{formula, R tall} represents a portrait _{of, R Topheavy} represents a superior _{degree, R Rightbiased} represents the right length _{of, L height} indicates the distance from the top of the finger ranging to the lower _{end, L width} is Indicates the distance from the right end to the left end of the finger range, L _upper indicates the distance from the upper end of the finger range to the reference point, L _lower indicates the distance from the lower end of the finger range to the reference point, and L _right indicates the distance of the finger range. The distance from the right end to the reference point is indicated, and L _left indicates the distance from the left end of the finger range to the reference point.

<About image features>
Next, an image feature amount acquisition method according to the present embodiment will be described with reference to the drawings. 9A to 9C are diagrams illustrating an example of data serving as a reference for acquiring image feature amounts. 10A and 10B are diagrams illustrating examples of ridge line information extraction results.

  In this embodiment, the use of contour information of hand images, which has been used for estimation in the conventional method, greatly affects individual differences. For example, the distance of each pixel of the hand image that is the foreground image from the background image is determined. Considering the height, the hand image is regarded as a mountain as shown in FIGS. In this embodiment, information on the ridge line that can be drawn on the mountain is used for estimation.

  Note that the point cloud shown in FIGS. 9A to 9C is obtained, for example, by contour scanning of a hand image. For example, when scanning the contour line of a hand image, a label “1” is attached to the pixel examined in the first scan, and after one round, the second contour scan is performed with the scanned pixel as the background, and the examined pixel Label “2” on A label “n” is attached to the pixel examined in the nth scan by the same process. The point cloud of FIGS. 9A to 9C is drawn with the label number attached by such processing as the height. However, the method for obtaining the ridge line information in the present invention is not limited to the above-described processing.

  As a result, as shown in FIGS. 10A and 10B, it is possible to reduce the influence of individual differences in the shape of the hand and the shape of goo, choki, par, etc., and increase the accuracy of finger shape estimation.

  Specifically, in the present embodiment, the finger image is cut out only in the finger range determined at the time of calculating the image shape ratio, and is reduced to 64 [pixel] and 64 [pixel]. When ridge line information is extracted from this reduced image, the finger image becomes an image as shown in FIG. 10B.

  In the method of extracting the ridge line, first, scanning is performed by tracing the outermost contour line of the hand image. At this time, the left and right pixels are examined in the scanning direction. If each pixel has the same pixel value as the background or has been scanned, the same pixel in the 64 × 64 [pixel] image prepared for the ridge line extraction result. Plot the point (corresponding to the ridge).

  When the contour line has been scanned once, the scanned pixel is regarded as the background, and the hand image contour line (one inner contour line) that is smaller by one is scanned, and the same processing as described above is performed. . When there are no more pixels to be scanned, the process ends. The ridge line image obtained by such an extraction process is divided into 8 vertical parts and 8 horizontal parts, and 25-dimensional image feature quantities are obtained using a high-order autolocal correlation function in each divided region.

  Here, FIG. 11 is a diagram for explaining high-order local autocorrelation processing performed on the ridge line information of each divided region of the 8 × 8 divided ridge line image. In the present embodiment, as shown in FIG. 11, the ridge line image is divided into 8 blocks (8 BLOCKS) in each of the vertical and horizontal directions, and 25 dimensions (No. 1 to No. 25 shown in FIG. 11) for each divided block. Are assigned as image feature amounts. Thereby, it is possible to obtain an image feature amount of 8 × 8 × 25 dimensions per one ridge line image.

  Here, with regard to the scanning of tracing the contour line of the hand image described above, the contour line scanning can be efficiently realized by performing pixel movement by the following method. 12A to 12C are diagrams illustrating an example of a pixel moving method.

  For example, in a 3 × 3 pixel matrix, when the traveling direction (scanning direction) is directly above, directly below, or directly beside, as shown in FIG. 12A, the current pixel (current search pixel) and the background pixel are The advance pixel is determined based on the relationship, and the pixel is moved to the determined advance pixel. Similarly, as shown in FIG. 12B, when the traveling direction is oblique, similarly, as shown in FIG. 12B, the pixel to be advanced is determined based on the relationship between the current pixel and the background pixel, and the pixel is moved to the determined pixel to be advanced.

  In addition, as an exception to the pixel moving method when the traveling direction (scanning direction) is oblique, the pixel can be moved using the position of the previous search pixel. Specifically, as illustrated in FIG. 12C, the advance pixel is determined based on the previous search pixel, the current pixel, and the background pixel, and the pixel is moved to the determined advance pixel. As a result, it is possible to determine the scheduled pixel with higher accuracy. Note that the pixel moving method illustrated in FIG. 12C can be applied to other oblique traveling directions, for example. Furthermore, the moving method shown in FIG. 12C can be applied to the above-described cases of directly above, directly below, directly beside, and obliquely. Accordingly, the traveling direction can be determined with high accuracy, and it is not necessary to scan the entire image, and only the scanning in the vicinity of the contour line is required, so that more rapid and efficient processing can be realized.

<Processing procedure for acquiring ridge line vector from contour scanning: other example>
Next, another example of acquiring the image feature amount from the ridge line will be described. Specifically, a processing procedure for acquiring a ridge line vector from contour scanning in the present embodiment will be described with reference to a flowchart. FIG. 13 is a flowchart illustrating an example of a processing procedure for acquiring a ridge line vector from contour scanning in the present embodiment.

  In FIG. 13, first, the image analysis unit 16 performs contour scanning on pixels other than the background or scanned pixels in the input image (S21), and determines whether or not the current pixel is a pixel on the ridge line (S22). . If the current search pixel is a pixel on the ridge line (YES in S22), the image analysis unit 16 determines whether there is a vector end point around the pixel (S23). If there is an end point of the vector around the pixel (YES in S23), the image analysis unit 16 then updates the vector end point to the current pixel coordinates, and the vector inclination is a constant value (preset). It is determined whether or not the threshold value has changed (S24).

  In the process of S24, when the gradient of the vector changes by a certain value or more (YES in S24), the image analysis unit 16 sets the current pixel as the end point of the current vector or the start point and end point of the new vector (S25). .

  In the above-described processing in S23, when there is no vector end point around the current pixel (NO in S23), the image analysis unit 16 sets the vector start point and end point as the current pixel coordinates (S26). In the above-described processing of S24, when the gradient of the vector does not change by a certain value or more (NO in S24), the image analysis unit 16 updates the vector end point to the current pixel coordinates (S27).

  In the process of S22, when the current pixel is not a pixel on the ridge line (NO in S22), or after the process of any one of S25, S26, and S27 is finished, the image analysis unit 16 finishes the contour line scanning. Whether or not (S28). When the contour line scanning has not ended (NO in S28), the image analysis unit 16 moves the pixel to be searched (analyzed) to the next pixel (S29), and then returns to S22 to perform the subsequent processing. Do. In addition, as a moving method to the next pixel in this process, for example, the moving method shown in FIGS. 12A to 12C described above can be used, but the method is not limited to this.

  Further, in the process of S28, when the contour line scanning is completed (YES in S28), the image analysis unit 16 determines whether there is a pixel other than the background or the scanned pixel (S30), and the pixel is If there is any (NO in S30), the process returns to S21 to perform subsequent processing. If there is no such pixel (YES in S30), the image analysis unit 16 ends the process.

<Finger shape estimation method>
Next, the finger shape estimation method in the present embodiment will be specifically described. First, a finger region is obtained from an image obtained from the imaging device 21 such as a camera, and for example, as described above, an image shape ratio, an image feature amount, and the like are obtained. Next, a database search is performed. In this embodiment, a two-step database search method is used.

  In the search at the first stage, for example, narrowing down by the image shape ratio is performed using threshold values as shown in the following equations (4) to (6).

Here, in the above-mentioned _{formula, th tall} represents a threshold for Vertical _{degree, th Topheavy} represents a threshold for superior _{degree, th Rightbiased} represents a threshold for the right length _{of, R tall} [i] is the i th data set R _topheavy [i] indicates the upper length of the i-th data set, R _rightbiased [i] indicates the right length of the i-th data set, and R _current-tall indicates the vertical length of the input image. R _{current-topheavy} indicates the upper length of the input image, and R _{current-rightbiased} indicates the right length of the input image.

  Next, in the second stage search, similarity calculation based on image feature amounts is performed. For the similarity calculation, for example, a simple Euclidean distance or the like is used, and the similarity is calculated using, for example, Expressions (7) and (8).

Here, in the above equation, E is indicated similarity, e _k represents a degree of similarity in the k-th divided area, x. current [j] _k represents the image feature amount (first feature amount) of the pattern j of the input image, x. dataset [i] [j] _k represents the image feature amount (second feature amount) of the pattern j of the i-th data set. Note that i indicates a data set number, j indicates a HLAC (Higher Order Local AutoCorrelation) pattern number, k indicates a divided region number, D indicates the number of divided regions, and P indicates an HLAC pattern. Indicates a number.

  However, in the present embodiment, a two-stage database search method is not essential, and for example, narrowing down by the first-stage image shape ratio may be omitted. Alternatively, refinement by parameters other than the image shape ratio may be used in the first stage, and further, a database search method including three or more stages using other data is performed to perform finger shape estimation. You can also.

<About matching using vector information>
Here, when the above-described vector information is used to perform the collation processing by the above-described collation unit 17 or the like, for example, a data set including at least one of the vector feature amount and the number of vectors is necessary. It is preferable to prepare in advance a database that is accumulated in a number.

  Further, as the first stage process of the collation process, the database is narrowed down. Specifically, for example, when the number of ridge line vectors extracted from a finger image captured from the imaging device 21 such as a camera is compared with the number of vectors in all data sets in the database, the difference in the number of both is a constant value. Choose one within.

  Next, as the second stage process, the most similar data set is selected from the data set group selected in the first stage process. Specifically, the most similar data set is selected using a vector feature amount from the data set group narrowed down in the first stage process. Here, a process flowchart in the second stage process will be described with reference to the drawings.

  FIG. 14 is a flowchart illustrating an example of the procedure of the second stage in the matching process. In FIG. 14, the collation unit 17 first selects the i-th data set from the data set group narrowed down by the first stage process (S41). Next, the matching unit 17 refers to the vector jth start point obtained from the camera image (S42).

Next, the collation unit 17 searches the vector feature amount in the i-th data set for the start point of the coordinate closest to the reference start point coordinate (S43), and the vector extending from the reference start point is determined in the previous step. The angle Angle _ij formed with the vector extending from the selected starting point and the length difference Length _ij between the two vectors are examined (S44).

  Here, the collation unit 17 determines whether or not there is already a starting point (S45). If there is a starting point (NO in S45), the variable j is incremented (j ++) (S46), and the process returns to S42. The unit 17 performs the subsequent processing. If there is no start point in the processing of S45 (YES in S45), the collation unit 17 then uses the following formulas (9) and (9) as the total number of vector start points of the hand image obtained from the camera. 10) is calculated (S47).

Next, the collation unit 17 sets the sum of normalized SumAngle _i and SumLength _i as similarity E _i (S48). Next, the collation unit 17 sets E _min as the provisional minimum similarity, and if E _min > E _i , substitutes E _i for E _min and temporarily sets the data set in which the i-th data set is most similar. (S49).

  Here, the collation unit 17 determines whether or not there is a data set to be examined (S50). If there is a data set to be examined (YES in S50), the variable i is incremented (i ++) (S51), and thereafter The collation unit 17 returns to S41 and performs subsequent processing. If there is no data set to be examined (YES in S50), the collation unit 17 finally outputs angle information and the like of the most similar data set (S52), and the process ends. In addition, about the output content in this embodiment, it is not limited to angle information, For example, you may output the other information contained in a data set, or all the information of a data set.

<Evaluation experiment>
Here, for example, in a finger shape estimation device in which the user's finger shape is registered, a user with a thick finger is created by wearing a work gloves on the designer, and a change in estimation accuracy due to individual differences is evaluated. Here, FIGS. 15A and 15B are diagrams illustrating an example of the difference between the contour line information and the ridge line information obtained when the finger shape is the same between the bare hand state and the military hand state. FIG. 15A shows the contour line information and ridge line information in the state of bare hands, and FIG. 15B shows the contour line information and ridge line information in the state of wearing gloves.

  In the evaluation experiment, using the same input data, the evaluation results were compared by comparing the estimation results of the conventional finger shape estimation system using contour information with the estimation results of the system using the ridge line information of this method. went. Also, in the evaluation experiment, without putting on the right hand, place it in the space where the camera (for example, View PLUS (with registered trademark) MV wide-angle lens) is installed, wear a data glove on the left hand, The input image and the joint angle data at that time were created as input data by simultaneously moving the left hand and the left hand.

  Further, in the evaluation experiment, as an example, the experiment was performed focusing on the grasping operation and the pinching operation with the forearm rotation angle fixed at 180 degrees. Here, FIG. 16A-16C is a figure which shows the comparison result of a joint angle. FIG. 16A shows a graph comparing the angle of the interdigital joint (interphalangeal joint) when the forearm rotation angle is 180 degrees, and FIG. 16B shows the index PIP joint (proximal interphalangeal joint). The graph which compared the angle is shown, FIG. 16C shows the graph which compared the little finger PIP joint angle. In FIG. 16A to FIG. 16C, the horizontal axis indicates the time (TIME [NUMBER OF FRAME]) obtained from the time-series frame of the captured image, and the vertical axis indicates the joint angle (JOINT ANGLE [DEGREE]). Show. FIG. 17 is a diagram showing an example of error average and error standard deviation.

  In the examples of FIGS. 16A to 16C, the actual value (MEASURED), the value obtained using the contour line (ESTIMATED BY OUTLINE), and the value obtained using the ridge line in the present embodiment (this method) ( ESTIMATED BY RIDGE LINE). In the example of FIG. 17, the error average [degree] and the error standard deviation [degree] in the thumb IP joint angle, the index finger PIP joint angle, and the little finger PIP joint angle corresponding to FIGS. 16A to 16C are shown.

  As shown in FIG. 17, when a person who has not registered a finger shape in the database uses the system, an estimation method using contour line information as in the conventional method and an estimation method using ridge line information (this When the forearm rotation is 180 degrees, the average estimation error range of the thumb IP joint angle is 0.33 ± 14.97 degrees (conventional technique) to −0.10 ± 13. 78 degrees (this method). The average estimation error range of the index finger PIP joint angle is changed from 0.99 ± 17.20 [degree] (conventional method) to −0.04 ± 16.89 [degree] (present method), and the little finger PIP The range of the average estimation error of the joint angle is 5.57 ± 20.33 [degrees] (conventional method) to 7.84 ± 15.45 [degrees] (present method).

  That is, by using the ridge line information as in this method, the estimation accuracy can be increased without adding a data set corresponding to individual differences. Therefore, by applying this method, many people can use the finger shape estimation system more comfortably.

<Application example of finger shape estimation technique of the present invention>
Here, an application example of the finger shape estimation technique of the present invention will be described with reference to the drawings. 18A and 18B are diagrams showing application examples of the present invention. The finger shape estimation technique of the present invention can be applied to the operation of a remote robot, for example, as shown in FIG. 18A.

  In this application example, the actual movement of the user's finger 42 of the user 41 is imaged by the imaging device 21 such as a camera at the terminal (hand shape estimation device 10) on the user (operator) side of the robot 40 in a remote place. To do. Then, by performing the above-described finger shape estimation processing of the present invention from the captured image (here, video), the actual movement of the user's finger 42 is estimated with high accuracy, and the estimation result is transmitted to the robot side. Send. Thereby, the same movement as the finger 42 of the user 41 is performed on the finger 43 of the robot 40, and the robot 40 can be remotely operated.

  Note that, as shown in FIG. 18A, an image or the like captured by the robot camera 44 can be displayed on the screen of the finger shape estimation device 10. Then, the user 41 can cause the robot 40 to perform a predetermined movement while watching the operation of the finger 43 of the robot 40 displayed on the screen. In the example of FIG. 18A, for example, a finger motion may be estimated using a database of gripping motions as shown in FIG. 7B described above.

  The example shown in FIG. 18B is an example in which the finger shape estimation technique of the present invention is realized using the camera function provided in the portable terminal 50 or the like. In this case, it is possible to estimate the shape and movement of the finger 52 of the user 51 at high speed and with high accuracy without wearing sensors or without a dedicated controller. Here, the portable terminal 50 corresponds to the finger shape estimation apparatus 10 in the present embodiment described above.

  Therefore, in this case, for example, a desktop metaphor (desktop environment) driven by a finger operation, a cloud computer or a remote conference system driven by a finger operation in which the mobile terminal 50 is combined with the mobile projector function 53, etc., three-dimensional by a finger operation The finger shape estimation technique of the present invention can be used for computer input of modeling information, virtual game by gesture gesture, home appliance operation without a controller, remote robot control without a dedicated control device, and the like.

  As described above, according to the present invention, the shape and movement of fingers in an image can be estimated at high speed and with high accuracy. Specifically, the present invention enables operations of information devices, home appliances, robots, and the like by moving fingers and arms in the same way as daily operations without wearing sensors. In other words, for example, a new desktop manager (desktop metaphor) that can operate a personal computer by daily operations such as opening a document or rolling a document into a trash can instead of entering information with a keyboard or mouse. Can be realized.

  Further, according to the present invention, a complicated free-form three-dimensional information input such as clay work is input by a finger operation instead of using CAD (Computer Aided Design) software, and a free-form three-dimensional object is input. Can be shaped. In addition, home appliances such as televisions and video devices can be controlled by finger operation without a remote control box, and remote operation of the robot can be performed by daily operation instead of a dedicated controller.

  That is, according to the present invention, since the matching data set is created using information that does not affect individual differences, the feature quantity dimension of the data set can be added without adding a special data set for individual differences. Therefore, the contents of the data set can be enriched, and finger shape estimation can be performed at high speed and with high accuracy. In order to enrich the contents of the data set, for example, in the past, for example, the data set was accumulated every 45 degrees with respect to the refraction angle of the finger, and created at intervals of 5 degrees or 10 degrees. Means that. That is, according to the present invention, the state in which each finger joint is bent halfway can be accumulated in multiple stages, whereby the finger shape can be estimated with high resolution and high accuracy.

  As mentioned above, although the preferable example of the finger shape estimation technique of this invention was explained in full detail, this invention is not limited to the specific embodiment which concerns, and in the range of the summary of this invention described in the claim Various modifications and changes are possible.

2. Second Embodiment In the second embodiment, a nail region extraction technique (a nail region extraction device, a nail region extraction method, and a nail region extraction program) for extracting a nail region in a finger image will be described.

<Conventional nail region extraction technology>
Conventionally, as a technique for specifying the tip position of a finger, for example, Japanese Unexamined Patent Application Publication No. 2009-265809 discloses a technique for adding nail presence information to information at the time of estimation to improve estimation accuracy. In this document, a classifier is constructed by machine learning using a database that collects feature values of nail region pixels in advance and a database that collects feature values of skin region pixels that do not include a nail, and the classifier is used. By detecting the nail, the movement of the fingertip or the like is identified, and the control assigned to each movement is executed.

  However, in the technology as shown in the above document, it is necessary to manually cut out the nail and analyze it when creating the database. When creating a database for each individual, a great deal of time and effort is spent for each individual. I had to. Moreover, this technique only performs pixel region determination by using a discriminator. Therefore, for example, when only the areas where the color of the pixel of the nail area and the skin area generally differ greatly, such as a fingertip, can be discriminated, for example, the color existing on the palm side is very An image containing skin region pixels similar to a nail could not be handled. In addition, since hands have different colors for each person, it is difficult to cope with individual differences.

  In the second embodiment, the above-described problem is solved, and a nail region extraction technique (nail region extraction device, nail region extraction method, and nail region extraction) that can extract a nail region in a finger image with high accuracy is provided. Program).

<Overview of Second Embodiment>
The nail region extraction device according to the second embodiment is a nail region extraction device that extracts a nail region included in an image captured by an imaging device, and an image acquisition unit that acquires an image captured by the imaging device And an image obtained by the image acquisition unit, and a nail region extraction unit that extracts a nail region from a predetermined feature amount obtained from the analysis result, the nail region extraction unit A separation plane is generated using only the obtained color information, a nail region candidate is extracted based on the generated separation plane, and pixel determination using an image including a preset palm is performed on the nail region candidate. The nail region is re-determined.

  A nail region extraction method according to the second embodiment is a nail region extraction method for extracting a nail region included in an image photographed by an imaging device, and an image for acquiring an image photographed by the imaging device. An acquisition step; and an nail region extraction step of analyzing the image obtained by the image acquisition step and extracting a nail region from a predetermined feature amount obtained from the analyzed result, wherein the nail region extraction step includes: A separation plane is generated using only color information obtained from the image, a nail region candidate is extracted based on the generated separation plane, and the nail region candidate is obtained by pixel discrimination using an image including a preset palm. The nail region is re-determined with respect to.

  Furthermore, a nail region extraction program according to the second embodiment is a nail region extraction program for implementing each processing step of the nail region extraction method on an information processing apparatus.

  According to the nail region extraction technique according to the second embodiment described above, the nail region in the image can be extracted with high accuracy.

<About the nail | claw area | region extraction technique which concerns on 2nd Embodiment>
In the nail region extraction technology according to the second embodiment, pixel generation is performed while performing separation plane generation using only color information and pixel determination for enabling nail region estimation even for an image including a palm. Re-determine later. In other words, in the present embodiment, by having the process (algorithm) for determining the separation plane, each analysis of the nail and the skin becomes unnecessary, so that the processing content can be reduced and the processing speed can be improved. . Further, in this embodiment, for example, unlike the conventional method, since estimation using a database or the like is not performed, it is possible in principle to cope with the influence of individual differences. Furthermore, in this embodiment, since a method for removing a skin region having a color similar to the color of the nail is provided, the nail region can be extracted with high accuracy for the entire finger region.

  Hereinafter, a nail region extraction device, a nail region extraction method, and a nail region extraction program according to the present embodiment will be described with reference to the drawings.

<Nail region extraction device: functional configuration example>
First, a functional configuration example of the nail region extraction device in the present embodiment will be described with reference to the drawings. FIG. 19 is a diagram illustrating an example of a functional configuration of the nail region extraction device according to the present embodiment. 19 includes an input unit 111, an output unit 112, a storage unit 113, an image acquisition unit 114, an image analysis unit 115, a nail region extraction unit 116, and a finger shape estimation unit 117. And a transmission / reception unit 118 and a control unit 119.

  The input unit 111 accepts input such as start / end of various instructions such as an image acquisition instruction, an image analysis instruction, a nail region extraction instruction, a finger shape estimation instruction, and a transmission / reception instruction from a user or the like. Note that the input unit 111 includes a pointing device such as a keyboard and a mouse if the nail region extraction device 110 is a general-purpose computer such as a PC (Personal Computer), and is an information terminal device such as a smartphone or a mobile phone or a game device. If there are, it consists of each operation button group. Further, the input unit 111 may have a voice input function for inputting voice such as the above-described instructions by voice or the like.

  The output unit 112 outputs information such as the content input by the input unit 111 and the content executed based on the input content. Specifically, the output unit 112 displays a screen display and a sound output of processing results and processing progress of each component in the nail region extraction device 110 such as an acquired image, an image analysis result, a nail region extraction result, and a finger shape estimation result. Etc. The output unit 112 includes a display, a speaker, and the like. Further, the output unit 112 may have a printing function such as a printer, and the above-described output contents can be printed on various printing media such as paper and provided to a user or the like.

  The accumulation unit 113 accumulates various types of information necessary in the present embodiment, various types of data at the time of execution of processing or after execution of processing, and the like. Specifically, the storage unit 113 is an image stored in advance, an image obtained by shooting or the like acquired by the image acquisition unit 114 (for example, including a time-series image such as a video), and the like. Accumulate. The accumulation unit 113 accumulates analysis results analyzed by the image analysis unit 115, extraction results by the nail region extraction unit 116, estimation results by the finger shape estimation unit 117, and the like. Further, the storage unit 113 can read out various data stored as necessary.

  The image acquisition unit 114 acquires, for example, an image or video captured by the imaging device 120 or the like. For convenience of explanation, it is assumed that the image acquired by the image acquisition unit 114 includes a finger, but the present embodiment is not limited to this.

  Here, in the present embodiment, the imaging device 120 is provided outside the nail region extraction device 110. However, the present embodiment is not limited to this, and the imaging device 120 is, for example, in the nail region extraction device 110. It may be built in. In addition, the images and videos acquired by the image acquisition unit 114 are not limited to images and videos of actual fingers captured by the imaging device 120. For example, a model of fingers, photos, posters, and the like were captured. It may be an image or the like. The image acquisition unit 114 can also acquire images, videos, and the like stored in an external device or database connected on the communication network via the transmission / reception unit 118. The image acquired by the image acquisition unit 114 can be stored in the storage unit 113 and can be read out from the storage unit 113 as necessary.

  The image analysis unit 115 analyzes the image acquired by the image acquisition unit 114. Specifically, the image analysis unit 115 indicates in which part (position, region) the position of an object such as a finger or a nail is projected from the feature amount of each pixel in the image, or the finger in the video. And how objects such as nails are moving. That is, the image analysis unit 115 digitizes the feature amount of the image of the captured hand or nail.

  The nail region extraction unit 116 extracts nail region candidates included in the image based on the result analyzed by the image analysis unit 115. Note that the nail region to be extracted can be extracted based on, for example, luminance information of an image, a threshold value, or the like. Further, the nail region extraction unit 116 can obtain the center-of-gravity coordinates or center coordinates of each nail from the extracted nail region and output it as position information. However, the present embodiment is not limited to this. For example, at least one of the nail presence information, the center of gravity, and the area area for each nail may be output. A specific nail region extraction method in the nail region extraction unit 116 will be described later. Information about the extracted nail region can be accumulated in the accumulation unit 113 and can be read out from the accumulation unit 113 as necessary.

  The finger shape estimation unit 117 estimates the finger shape based on the information on the nail region set by the nail region extraction unit 116. Specifically, using a nail position information, a finger contour shape (contour line information), and the like included in the image, a database in which the finger shape corresponding to the nail position information and the finger contour shape is set in advance is input. By collating with the image, the shape of the finger can be estimated with high accuracy. By using nail information as shown in the present embodiment, for example, it is possible to determine with high accuracy whether the finger shape is the back side of the hand or the palm side. In addition, information on how much the palm or back of the hand is rotating with respect to the camera or the like can be estimated with high accuracy. In the present embodiment, the finger shape estimation unit 117 may not be provided.

  The transmission / reception unit 118 also includes an external image desired from an external device that can be connected using a communication network or the like, an execution program for realizing the nail region extraction processing in the present embodiment, and the like. It is an interface for obtaining. Further, the transmission / reception unit 118 can transmit various information obtained in the nail region extraction device 110 to an external device.

  The control unit 119 controls the entire components of the nail region extraction device 110. Specifically, the control unit 119 performs control in each process such as image acquisition, image analysis, nail region extraction, finger shape estimation, and the like based on, for example, an instruction from the input unit 111 by a user or the like. .

  The imaging device 120 includes a digital camera, a high-precision camera, and the like, and acquires images and videos of the user's actual fingers and model fingers. Note that only one imaging device 120 may be provided, or a plurality of imaging devices 120 may be provided so that fingers can be photographed simultaneously from different directions.

<Nail region extraction device 110: hardware configuration>
Here, the above-described nail region extraction device 110 generates an execution program (for example, a nail region extraction program) as software that can cause a computer (hardware) to execute each function, and is a general-purpose device such as a PC. By installing the execution program in an information terminal device such as a personal computer, a server, a smartphone or a mobile phone, a game machine, or the like, the nail region extraction processing or the like in the present embodiment can be realized.

  Here, a hardware configuration example of a computer capable of realizing the nail region extraction process in the present embodiment will be described with reference to the drawings. FIG. 20 is a diagram illustrating an example of a hardware configuration capable of realizing the nail region extraction process according to the present embodiment.

  20 includes an input device 121, an output device 122, a drive device 123, an auxiliary storage device 124, a memory device 125, a CPU (Central Processing Unit) 126 that performs various controls, and a network connection device. 127 are connected to each other by a system bus B.

  The input device 121 has a pointing device such as a keyboard and a mouse operated by a user or the like, and inputs various operation signals such as execution of a program from the user or the like. Further, the input device 121 may include an image input unit that inputs an image taken from the imaging device 120 such as a camera.

  The output device 122 has a display for displaying various windows and data necessary for operating the computer main body for performing processing in the present embodiment, and the execution program and results of the program are displayed by the control program of the CPU 126. Can be displayed.

  Here, the execution program installed in the computer main body in the present embodiment is provided by a portable recording medium 128 such as a USB (Universal Serial Bus) memory or a CD-ROM, for example. The recording medium 128 on which the program is recorded can be set in the drive device 123, and the execution program included in the recording medium 128 is installed in the auxiliary storage device 124 from the recording medium 128 via the drive device 123.

  The auxiliary storage device 124 is a storage device such as a hard disk, and can store an execution program in this embodiment, a control program provided in a computer, and the like, and can input and output them as necessary.

  The memory device 125 stores an execution program read from the auxiliary storage device 124 by the CPU 126. The memory device 125 includes a ROM (Read Only Memory), a RAM (Random Access Memory), and the like.

  The CPU 126 controls processing of the entire computer, such as various operations and input / output of data with each hardware component, based on a control program such as an OS (Operating System) and an execution program stored in the memory device 125. Thus, each process in the nail region extraction process can be realized. Various information necessary during the execution of the program can be acquired from the auxiliary storage device 124, and the execution result can be stored in the auxiliary storage device 124.

  The network connection device 127 obtains an execution program from another terminal connected to the communication network by connecting to a communication network or the like, or an execution result obtained by executing the program or in the present embodiment. The execution program itself can be provided to other terminals.

  With the hardware configuration as described above, the nail region extraction process in the present embodiment can be executed. Also, by installing the program, the nail region extraction process in the present embodiment can be easily realized by a general-purpose personal computer or the like.

  Next, the nail area extraction process in the above-described nail area extraction program will be specifically described.

<Nail region extraction processing procedure>
First, an outline of the nail region extraction processing procedure in the present embodiment will be described. FIG. 21 is a flowchart illustrating an example of a nail region extraction processing procedure in the present embodiment. Note that the operation of each unit in various processes described below is controlled by the control unit 119 (CPU 126).

  In the nail region extraction process shown in FIG. 21, first, the image acquisition unit 114 acquires an image captured by the imaging device 120 such as a camera (S101). Next, the image analysis unit 115 analyzes the image (S102), and acquires position information of objects such as fingers and nails included in the image.

  Next, the nail region extracting unit 116 extracts a nail region based on the information obtained in the process of S102 (S103). Next, the finger shape estimation unit 117 estimates the finger shape based on the extracted nail region and the like (S104). Then, the finger shape estimation unit 117 outputs the estimation result (S105). In the present embodiment, the present invention is not limited to this. For example, the finger shape estimation unit 117 may output only the nail region in the image after the process of S103 ends.

  Next, the control unit 119 determines whether or not to end the process (S106). If the process does not end (NO in S106), the control unit 119 returns to S101, and the control unit 119 performs, for example, continuous images, that is, videos. Then, the control unit 119 performs the above-described processing by performing the above-described processing and outputting the result in time series or acquiring another image.

  Further, in the process of S106, when the process is terminated by a user instruction or the like (YES in S106), the control unit 119 ends the nail region extraction process.

<S103: Nail Region Extraction Processing>
Next, a specific example of the nail region extraction process in S103 described above will be described with reference to the drawings.

[Distribution of pixels constituting the fingers in the image]
First, the characteristics of the pixel distribution of the finger image in the RGB color space will be described. FIG. 22 is a diagram illustrating an example of a pixel distribution model in the RGB color space of a finger image. As shown in FIG. 22, the skin area pixels (SKIN AREA DISTRIBUTION) are densely distributed like a thin ellipsoid, and the nail area pixels (NAIL AREA DISTRIBUTION) are part of the space above the ellipsoid. It is distributed in such a way that it rides in layers while sharing. Here, this shared part (COMMOM AREA) increases as the lightness decreases.

  Therefore, basically, brightening the entire finger image is an important condition for separating the nail region pixel distribution from the skin region pixel distribution. However, on the other hand, since light reflection differs depending on the shape of the hand and the positional relationship between the camera and the hand, the brightness (luminance, etc.) may decrease depending on the location and the like, and the shared portion may increase. In such a case, it is impossible to cope with only the above-mentioned conditions. Therefore, in order to accurately detect the nail from various hand shapes, it is necessary to consider the difference in brightness due to this difference in reflection.

[Nail region extraction method in the present embodiment considering brightness difference]
Therefore, in this embodiment, the nail region is extracted based on the following method. FIG. 23 is a flowchart showing a specific example of nail region extraction processing in the present embodiment. Moreover, although the background of a finger image is black, it is not limited to this.

  First, the nail region extraction unit 116 performs preprocessing using, for example, a median filter in order to remove imaging noise from the input image (S111). The preprocessing includes, for example, smoothing processing, but is not limited to this in the present embodiment. Further, the preprocessing may include processing such as contrast adjustment and background separation. In the smoothing process by the pre-processing described above, noise removal is performed by using a median filter such as 3 × 3 which is a small framework in order to remove, for example, imaging noise. Further, since the purpose is to remove imaging noise, in the smoothing process, for example, a non-linear filter (that is, a filter that is not affected by an abnormal value) is used, but the present invention is not limited to this.

  Next, the nail region extraction unit 116 extracts pixels similar to the color of the nail (S112), and performs a binarized image smoothing process and a labeling process to thereby extract an area having a color similar to the nail. It is detected as a nail region candidate (S113). In the smoothing process described above, for example, a median filter such as 7 × 7 is used in order to combine pixels extracted separately into one region, but the filter region is limited to 7 × 7. It is not something. In addition, the smoothing process by the median filter, specifically, first divides the original image into three images of an R channel image, a G channel image, and a B channel image, and then smoothes each image. Then, the three images after smoothing are again integrated into one RGB image. In this embodiment, since the purpose is to combine pixels, a median filter is used as a smoothing method. However, in the present embodiment, the present invention is not limited to this. For example, a weighted average filter, a Gaussian filter, or the like is used. The smoothing method can be used.

  In the labeling process described above, for example, in order to cause a computer to recognize a region and acquire the center of gravity of the region, the regions appearing by smoothing are numbered in the order of the size of the region. A small area (specifically, for example, an area having 20 or less pixels) is regarded as noise and removed. Thereafter, the position of the center of gravity of each remaining region is acquired.

  Since the region obtained by the above-described processing is a region obtained from the entire finger image, there is a possibility that it may be affected by a difference in light reflection due to imaging as described above.

  Therefore, in this embodiment, the nail region extraction unit 116 further sets a ROI (Region Of Interest) around the center of gravity of the nail candidate region (S114), and performs reprocessing within the ROI (S115). Note that the reprocessing is the above-described nail region extraction processing, and for example, shows the processing from S111 to S113 described above, but is not limited to this in the present embodiment. By performing the reprocessing in S115, it is possible to reduce the influence or the like due to the difference in light reflection.

  Thereafter, the nail region extraction unit 116 determines the final nail region by determining again whether the obtained nail region is a nail (S116), and acquires and outputs the barycentric position of the nail region ( S117). Thus, in the present embodiment, the nail region is extracted in consideration of the difference in brightness.

<S112: Method for Extracting Pixels with Color Similar to Nail>
Next, a method for extracting a pixel having a color similar to a nail in the process of S112 described above will be described. 24A to 24C are diagrams illustrating an example of coordinate conversion based on the principal component axis. 24A shows the RGB pixel distribution of finger images of skin (SKIN) and nails (NAIL), and FIG. 24B shows the first principal axis (1ST MAIN AXIS) -third principal axis (3RD MAIN AXIS) of skin and nails. FIG. 24C shows a pixel distribution in the second principal axis (2ND MAIN AXIS) -third principal axis plane of the skin and nails. 25A and 25B are diagrams illustrating an example of a separation plane position determination method. Here, as an example, the first main axis means a lightness axis in skin color, the second main axis means a warm color axis, and the third main axis means a cold color axis. Is not limited to this.

In the present embodiment, first, coordinate conversion is performed on the color information of the finger image excluding the background with a principal component axis vector obtained by eigenvalue decomposition of the variance-covariance matrix as a basis. At this time, as shown in FIG. 24A, layers of nail pixel distribution and skin pixel distribution appear in a direction perpendicular to the third principal axis. Therefore, the equation of the separation plane to be obtained is expressed as follows when the coordinates of the pixel after coordinate conversion are x = (x ₁ , x ₂ , x ₃ ) ^T (where T is a transposed matrix): 11) and can be expressed in a very simple one-dimensional form.

  Here, the threshold Thread_layer is a value when a palm-only image without a nail is used as an input image as a calibration at the start of the system. The reason for using a palm-only image is that the palm generally has a skin color close to white compared to the back side of the hand, so the brightness of the pixels is high, and it resembles the color of the nail. This is because the pixels are distributed in the upper part.

In other words, if calibration is performed on an image showing only the back side of the hand, the separation threshold is lowered, and many skin pixels on the palm side are extracted, which is not suitable. Therefore, in the present embodiment, for example, only the dense portion of the pixel distribution of the palm-only image i is extracted, the variable Thread_layer _i is moved from the top to the bottom as shown in FIG. 25A, and the straight line in Expression (11) is the dense pixel. The position in contact with the area is set as the value of Thread_layer _i . Then, Thread_layer is defined as the following equation (12).

Here, in the above equation (12), n is the number of images used for calibration, and off _th is an offset constant for finely adjusting the position at which the layer is cut. By using these methods, it is possible to automatically determine an appropriate cutting position (position of the separation plane).

  That is, in the above-described processing, as shown in FIG. 25A, first, the principal component axis basis conversion is performed on the pixel information of the image in which only the palm is captured. After that, the position of the separation plane is lowered from a high position, and the point where the separation plane is reached is defined as the third principal axis coordinate (3RD MAIN AXIS) of the separation plane. The average value of the position of the separation plane calculated by performing this calibration on a plurality of (n) images is set as the third principal axis coordinate of the separation plane used in this embodiment.

  Next, as shown in FIG. 25B, the finger image is principal component axis basis transformed to separate the nail pixel (NAIL PIXELS) and the skin pixel (SKIN PIXELS) from each other on the separation plane obtained by the above-described method, For example, an area having a color similar to a nail is obtained by a process such as smoothing or labeling.

  As described above, in this embodiment, a two-dimensional plane is created with two of the three axes obtained by principal component analysis, and the height of the separation plane is changed with one of the two created axes. . Specifically, in the present embodiment, the nail region extraction unit 116 sets pixel information of an image in which only the palm imaged in advance by the imaging device 120 is captured from the first to third values set in advance by principal component analysis. The principal component axis basis transformation is performed using two of the principal axes, the position of the separation plane is lowered from a high position along one principal axis of the two principal axes, and the place where the density area is reached A separation plane is used, and a nail region is extracted using the separation plane.

  26A and 26B are diagrams showing the extracted nail region portion in the present embodiment. In the present embodiment, nail region candidates as shown in FIG. 26B are extracted from the original image shown in FIG. 26A by the above-described analysis or the like. In addition, the white area | region shown to FIG. 26B is the part extracted as a nail | claw area | region candidate. In the present embodiment, at least one nail region may be extracted. If the nail region does not meet the above-described conditions, the nail region may be processed as not present in the image.

<Nail determination method using density difference>
Next, a method for determining a nail by determining a nail in the above-described processing of S116 after determining a nail region candidate will be described. When determining the nail area candidate, the difference in light reflection depending on the imaging direction of the camera affects the nail that only appears in a very small area, or the skin that is mistakenly extracted in the area larger than the nail area is the nail area candidate. It may have become. Therefore, for example, in a simple process such as a smoothing process or a closing process, the nail area disappears before the skin area, and the erroneous extraction area may not be removed. It should be noted that the location where the region is extracted greatly and is erroneously extracted is roughly determined.

  Here, FIG. 27 is a diagram illustrating an example of a skin distribution position with a high probability of erroneous extraction. For example, there are many false extractions as described above, such as the thumb ball (THENAR), finger pad (FINGER PULP), and finger side surface (FINGER SIDE) shown in FIG. 27, and the false extraction probability when the cutting position of the separation plane is low. The skin of the vicinity of the little finger ball (ANTITHENAR) and MP joint (Metacarpophalangeal; Metacarpophalangeal joint) increases.

Therefore, in the present embodiment, a process for removing the above-described part is performed. First, a square ROI (region of interest) is set around the center of gravity of the nail region candidate, and a pixel having a color similar to the nail is re-extracted for each ROI. When the number of finger pixels in the i-th ROI is n _i , the target area Square _i targeted at the time of re-extraction is defined as, for example, the following equation (13).

  Expression (13) indicates that the ratio of the number of finger pixels in all ROIs to the target area is constant at a constant a. In the present embodiment, the separation plane is moved for each ROI based on this target area, and re-extraction is performed with different threshold values. This re-extraction of the area is performed, for example, using only the peripheral information of the nail region candidate, so it is possible to determine which pixels have similar brightness and which are similar to the nail color, and as a result, reflect light during imaging. This can reduce the influence of the decrease in extraction accuracy.

  Here, FIGS. 28A and 28B are diagrams illustrating examples of execution results when the nail region is re-extracted and when the erroneously extracted skin region is re-extracted, respectively. In FIGS. 28A and 28B, the state of pixels after re-extraction is concentrated around the center of gravity in the nail region shown in FIG. 28A. On the other hand, it can be seen that the erroneously extracted skin region shown in FIG. Therefore, in the present embodiment, the nail region is determined using the difference in the state of crowding.

In the present embodiment, the nail is determined by counting the number of dense pixels and the number of pixels that are not dense and comparing the ratio of the number of pixels. For example, a binary image after pixel re-extraction in the i-th ROI is set as O _i, and the image is smoothed by a median filter or the like, and an image leaving only a dense region is obtained as a dense image C _i. It is defined as Then, the exclusive OR of the binary image O _i and the dense image C _i is taken by the following equation (14), and the value is defined as the sparse image S _i .

At this time, when the number of pixels extracted pixels in dense image C _i N _{c ^i,} the number of pixels extracted pixels in sparse image S _i was N _s ^i, equation showing the condition is a nail, for example, the following It is defined by (15), and it is determined whether or not it is a nail based on the formula (15).

  That is, in the present embodiment, as described above, ROI is set from the center of gravity of each nail region candidate, and re-extraction is performed by changing the position of the separation plane for each ROI so that the ROI region area is the same. In this embodiment, an image is generated for each ROI to prevent pixel overlap. The ROI shape may be a square or a circle, but the shape and size of the ROI are not particularly limited.

<About evaluation results>
Next, the evaluation experiment of the nail | claw area | region extraction result based on this embodiment mentioned above and the evaluation result are demonstrated using figures. 29A to 29C are diagrams for describing the evaluation results in the present embodiment. Note that FIG. 29A shows the probability (DETECTION PROBABILITY [%]) that is actually determined as a nail from nail region candidates on the back of the hand (BACK) and palm (PALM) for each finger (THUMB (thumb), INDEX (index finger), FIG. 29B shows an example obtained for each of MIDDLE (middle finger), RING (ring finger), PINKY (little finger)) and skin (SKIN). FIG. (DISTANCE [PIXEL]) is shown for each finger, and FIG. 29C shows an example of the nail region centroid and ROI of an image showing only the back side of the hand.

  In the evaluation experiment, a fluorescent lamp is used to illuminate the camera from the top to the bottom, and one of the two LED lights is illuminated from the top to the bottom, and on the other side from the bottom to the top. Use a finger image taken from a distance. For example, a Dragonfly Express (640 × 480 [pixel]) manufactured by Point Gray Research was used as the camera.

  As the images, 100 images including only the back side of the hand and 100 images including the palm side are used in total of 200 images. The evaluation is performed for each finger with respect to the nail detected as the nail region candidate and the probability that the erroneously extracted skin is determined as a nail by the nail determination method, and the Euclidean distance error between the correctly extracted nail centroid and the actual centroid. Evaluation was performed.

The evaluation experiment was performed by adjusting the offset constant off _th so that the probability that each fingernail is recognized as a nail region candidate is 95% or more for each finger. In this embodiment, as an example, the search range of ROI is 40 × 40 [pixel] and the threshold Thread _cs of the nail determination condition is 2.5. However, in the present embodiment, the present invention is not limited to this. Instead, these parameters can be arbitrarily set according to, for example, the size and angle of fingers in the image, the total number of pixels of the input image, the image size, and the like.

  First, FIG. 29A shows the probability that a nail detected as a nail region candidate and erroneously extracted skin will be recognized as a nail. According to FIG. 29A, the technique in the present embodiment is 10% or less of erroneous skin extraction, and the skin is hardly erroneously detected. Furthermore, it was shown that the nails can be detected with an accuracy of more than 90% except for the detection result of the thumb nail in the image showing only the back side of the hand. In addition, the skin site | parts finally determined erroneously were the finger pad and the finger | toe side surface. The main reason for misextracting the finger side surface is considered to be that most of the ROI area was a nail because the misextracted position was located in the immediate vicinity of the nail area.

  This can be improved by adding a process such as processing if the error is found between the center of gravity of the pixel and the ROI center obtained by re-extraction in the nail determination process (algorithm). In addition, with regard to the finger pad, it has been found that pixels are sometimes concentrated at the center.

  Next, FIG. 29B shows the Euclidean distance error between the extracted centroid and the actual centroid. According to FIG. 29B, the Euclidean distance error of the center of gravity of the nail other than the thumb in the image showing only the back side of the hand is less than 4 [pixel] on average, and the center of gravity of the nail can be obtained with a considerably small error from the resolution of the image. I understood. Note that the center of gravity error is larger in an image that only shows the back of the hand than the image that includes the palm side of each finger. The nail area and a small area on the side of the finger adjacent to that area are combined, and the center of gravity is the center of the nail. It is thought that it is because it shifts to the outside direction.

  Finally, let us consider the reason why the accuracy of the image showing only the back side of the thumb of the thumb is significantly reduced compared to other images. An image including the thumb on the back side of the hand is captured as shown in FIG. 29C. According to FIG. 29C, the center of gravity is almost the center of the nail except for the thumb, but it can be seen that the center of gravity is at the end of the nail. As a result of the analysis, a shadow with extremely low brightness was created at the lower part of the nail of the thumb, and the color changed, and only the skin on the upper part of the nail and the side of the finger was extracted. It turned out to be the cause.

  Further, as an evaluation result, since the center of gravity, that is, the center of ROI is at the end of the nail, the search range near the thumb finger captures more skin pixels than other nails. Furthermore, the skin region pixels to be captured are pixels on the finger side whose color resembles a nail. As a result, with the thumb, the result is greatly influenced by the finger side surface pixels included in the ROI, and it is considered that an erroneous estimation has occurred. As described above, it is considered that this is caused by a situation where a local brightness difference due to a shadow is generated in the ROI, and the determination accuracy is lowered.

  In the present embodiment, a nail region detection system that distinguishes the nail from the density difference is constructed by paying attention to the property that the dense state of pixels having colors similar to the nail are different inside and outside the nail contour and on the skin. As a result of the evaluation experiment, the skin area is misjudged only with a low probability of 10% or less, and when the shadow cannot be locally applied to the nail, only the nail can be detected with high accuracy exceeding 90%, and the center of gravity of the nail is detected. It was confirmed that the deviation can be obtained with an average of 4 [pixel] or less. Therefore, by using this embodiment, the nail region in the image can be extracted with high accuracy.

<Application example of nail area extraction technology>
Here, an application example of the nail region extraction technique in the present embodiment will be described with reference to the drawings. FIG. 30 is a diagram illustrating an application example of the nail region extraction technique of the present embodiment. FIG. 30 shows a finger shape estimation system 130 that includes the function of the nail region extraction device according to the present embodiment and the function of an existing finger shape estimation device that uses contour lines. Specifically, the finger shape estimation system 130 includes a camera 131 that is an imaging device, an outline information acquisition processing system 132, a nail information acquisition processing system 133, and a database collation unit 134. Here, the nail information acquisition processing system 133 corresponds to the nail region extraction device 110 described above.

  An image including a hand obtained from the camera 131 is output to the contour line information acquisition processing system 132 and the nail information acquisition processing system 133. The contour line information acquisition processing system 132 acquires a contour line (contour shape) of a finger from the acquired image, and outputs a feature amount (contour line information) of the contour line. As an example of acquiring a contour line, for example, a finger part and a background part can be separated from an image based on luminance difference information between adjacent pixels and the contour line of the finger part can be acquired. The embodiment is not limited to this.

  The nail information acquisition processing system 133 acquires the nail information such as the center of gravity and contour of the nail by performing the above-described processing on the image acquired from the camera 131.

  As shown in FIG. 30, the database verification unit 134 uses the contour line information and the nail information, the contour line feature amount and nail presence information, the center of gravity, the area area, etc. for various types of finger shapes, Are combined with the data in the database previously stored in the storage unit or the like, and the finger shape having the highest similarity is output as the estimated shape of the hand.

  Further, the database collating unit 134 collates the input data with data in the database that is stored in advance by combining the joint angle information, for example, specifies the joint angle of the finger, and outputs the angle data and the like. Note that the information output by the database collation unit 134 is not limited to angle data, and for example, the content of hand movement (for example, the type of gripping movement) may be output.

  Further, the processing in the contour line information acquisition processing system 132 and the processing by the database collation unit 134 may be included in the processing content of the finger shape estimation unit 117 of the nail region extraction device 110 described above, for example.

  In general, the estimation of the finger shape is a multi-joint structure, and since fingers move in a complicated manner, there are problems such as a complicated calculation and a weakness against self-shielding when a three-dimensional model is established. For this reason, a method of estimating a three-dimensional shape at high speed by collating two-dimensional image information with data in a database is used. However, since the method is based on the contour line information, for example, if the finger is bent toward the front of the camera, the tip position information of the finger may be lost, and the accuracy may be lost.

  Therefore, in this embodiment, the finger tip position information can be obtained by using the nail information acquisition processing system 133 as in the finger shape estimation system 130 shown in FIG. 30, and the estimation accuracy can be improved. . Note that the nail information acquisition processing system as described above can be applied to a finger shape estimation device as shown in, for example, International Publication No. WO2009 / 147904 filed by the present applicant.

  As described above, according to the present embodiment, the nail region in the image can be extracted with high accuracy. Specifically, in the present embodiment, the nail region can be extracted with high accuracy and in real time only by moving the hand so that it is reflected in the camera without an attachment such as an artificial nail. As this technology develops, it will be possible to accurately determine the tip position of a finger. For example, the recognition accuracy of a technology that makes a computer recognize a motion involving a complicated finger shape such as a sign language motion, or a wrist that does not specify a shape. An improvement in estimation accuracy when estimating various finger shapes including rotation can be expected.

  Further, although the color of the nail is different from that of the skin, it is not easy to accurately separate the nail region and the skin region. In fact, in the conventional method, it is impossible to make a separation judgment formula unless the pixel distribution between the nail region pixel and the skin region pixel is analyzed in advance. Furthermore, since the nail area is very small in the finger area, when a color similar to the nail is extracted and the area is obtained by performing smoothing processing and labeling processing, the nail area has a color similar to that of the nail. The skin area becomes a larger area. For this reason, there are many cases in which a skin area cannot be removed by a simple noise removal method such as a smoothing process or a closing process, making it difficult to extract a nail area with high accuracy. However, according to the present embodiment, the separation plane for separating the nail pixel distribution and the skin pixel distribution can be determined without analyzing the nail and skin distribution, and the skin area that could not be removed can be removed. .

  In the present embodiment, separation plane generation using only color information and nail redetermination after pixel discrimination for correspondence with an image including a palm are performed. In the prior art, it is necessary to create a feature quantity database for nail area pixels and a feature quantity database for skin area pixels in order to generate a discriminator for constructing a separation hyperplane that determines which area a pixel belongs to There is. At this time, it is necessary to manually separate the skin and the nail, so it is necessary to manually separate the nail and the skin to generate a processed learning image, and generate many learning images in order to provide accuracy. Took a lot of time and effort.

  On the other hand, in the present embodiment, the property that the nail region pixels are very small compared to the skin region pixels, and the pixel information of the entire finger image is coordinate-transformed with the principal component axis basis, the nail region pixels and the third principal axis direction A large layered bias occurs in the distribution formed by the skin region pixels, and the characteristic that both pixel distributions can be separated by a linear separation plane is utilized. Then, by performing principal component axis basis coordinate conversion on a palm-only hand image in which a nail is not captured as a calibration in advance and obtaining a coordinate value in the third principal axis direction on the upper surface of the skin pixel distribution by determination based on pixel density, It is possible to derive the equation of the separation plane that automatically separates the nail pixel distribution from the skin pixel distribution without manual operation. Thereby, in this embodiment, time and labor can be reduced significantly.

  In addition, the conventional technology is based on the assumption that nail information is used when operating an information terminal device like a touch panel, so only the back side of the hand with fewer skin areas with pixels similar in color to the nail area pixels is identified. It was considered as a target. Therefore, in the prior art, only the type determination of whether a pixel is a nail region or a skin region is performed by a separation hyperplane using a discriminator. For this reason, it cannot be handled on the palm side in which a region where pixels having colors similar to nails are concentrated in a part of the skin region. On the other hand, in this embodiment, the point of determining a pixel using the separation plane is the same as that in the conventional technique, but whether or not the region is really a nail region for each region (ROI) including the pixel after the determination. It was made possible to deal with images containing palms by judging again.

  That is, since this embodiment is a system that can be used without manually cutting out nail area pixels and skin area pixels in advance, according to this embodiment, it is possible to make adjustments with a little adjustment. The system can be used. Moreover, since it can respond also to the finger image containing a palm, the position of a nail | claw can be calculated | required correctly with respect to various finger shapes. Furthermore, in this embodiment, since a two-stage determination method based on determination based on the separation plane and re-determination after region extraction is used, the possibility of erroneous extraction is reduced.

  Furthermore, in the present embodiment, the position of the nail region is obtained without attaching an attachment such as an artificial nail, that is, the tip position of the finger is always accurately known only by arranging the camera so that the nail always appears in the image. It becomes possible. Therefore, according to the present embodiment, for example, when it is desired to refine the operation of a three-dimensional gesture interface that moves a hand movement as it is in a virtual three-dimensional space, or to detect the movement of a nail region without touching It can be used for a non-touch motion detection terminal, a sign language recognition device, etc. that can move the terminal as if it were touched.

  Furthermore, even if there are a plurality of hands in the image, the finger shape can be estimated with higher accuracy by extracting the nail region by the method of the present embodiment.

  As mentioned above, although the preferable Example of the nail | claw area | region extraction technique was explained in full detail, the nail | claw area | region extraction technique of this embodiment is not limited to the specific embodiment which concerns, A various deformation | transformation and change are possible.

  In the example illustrated in FIG. 30, the finger shape estimation system 130 that combines the nail region extraction device according to the second embodiment and the existing finger shape estimation device has been described. However, the nail according to the second embodiment is described. A finger shape estimation system (finger shape estimation device) may be constructed by combining the region extraction device (FIG. 19) and the finger shape estimation device (FIG. 1) according to the first embodiment.

  In this case, for example, the finger shape estimation device (FIG. 1) according to the first embodiment and the nail region extraction device (FIG. 19) according to the second embodiment may be combined as separate devices. Further, for example, even if a finger shape estimation system (finger shape estimation device) is constructed by incorporating the nail region extraction unit 116 in the nail region extraction device 110 shown in FIG. 19 into the finger shape estimation device 10 shown in FIG. Good.

  In the latter configuration, a component (for example, an input) that can be shared between the finger shape estimation device (FIG. 1) according to the first embodiment and the nail region extraction device (FIG. 19) according to the second embodiment. Unit, output unit, storage unit, image acquisition unit, image analysis unit, transmission / reception unit, control unit, etc.) may be shared by both devices. In this case, it is only necessary to control the operation of each component to be shared so as to support not only the finger shape estimation function but also the nail region extraction function. With such a configuration, not only the effects obtained in the first embodiment but also the same effects as the various effects described in FIG. 30 can be obtained.

DESCRIPTION OF SYMBOLS 10 Finger shape estimation apparatus 11,111 Input part 12,112 Output part 13,113 Storage part 14,114 Image acquisition part 15 Database construction part 16,115 Image analysis part 17 Collation part 18,117 Finger shape estimation part 19,118 Transmission / reception Unit 20, 119 Control unit 21, 120 Imaging device 31, 121 Input device 32, 122 Output device 33, 123 Drive device 34, 124 Auxiliary storage device 35, 125 Memory device 36, 126 CPU
37, 127 Network connection device 38, 128 Recording medium 40 Robot 41, 51 User 42, 43, 52 Finger 44 Robot camera 50 Mobile terminal 53 Mobile projector function 110 Nail region extraction device 116 Nail region extraction unit 130 Finger shape estimation system 131 Camera 132 Outline information acquisition processing system 133 Nail information acquisition processing system 134 Database verification unit

Claims (14)

An image acquisition unit for acquiring an image including a finger shape;
An image analysis unit that analyzes the image acquired by the image acquisition unit and acquires a first feature amount corresponding to a ridge line shape of a finger included in the image;
Based on the first feature value obtained by the image analysis unit, a reference database in which a second feature value corresponding to a predetermined finger shape set in advance is stored is referred to. A finger shape estimation unit for estimating a finger shape corresponding to the feature amount ,
The image analysis unit regards the image as one peak by regarding a finger image included in the image as a foreground, an image other than the finger image as a background, and a distance from the background image in the foreground image as a height. A finger shape estimation device that obtains information on the ridge line shape from the mountain-shaped image . Furthermore, a database construction unit for constructing the database is provided,
The finger according to claim 1, wherein the database constructed by the database construction unit stores at least angle data and the second feature amount corresponding to the predetermined finger shape with respect to the predetermined finger shape. Shape estimation device. The finger shape estimation unit narrows down a data set based on a predetermined shape parameter in the finger shape from the database, and performs similarity calculation on the narrowed data set group using the first feature amount. The finger shape estimation apparatus according to claim 1, wherein the most similar data set is output. The image analysis unit sets a start point and an end point of a ridge line vector based on a slope of the ridge line vector obtained by contour scanning of a finger image included in the image ,
The finger shape estimation unit according to any one of claims 1 to 3, wherein the finger shape estimation unit estimates the finger shape based on a position of a start point and / or an end point of a ridge line vector set by the image analysis unit. Estimating device. Further, a separation plane is generated using only color information obtained from the image acquired by the image acquisition unit, a nail region candidate is extracted based on the generated separation plane, and an image including a preset palm The finger shape estimation apparatus according to any one of claims 1 to 4 , further comprising a nail region extraction unit that performs redetermination of a nail region with respect to the nail region candidate by pixel determination using a nail . The nail region extraction unit performs nail determination using a property in which a pixel in the case of an actual nail and a pixel having a color similar to that of the nail are different in density between the nail region and the skin region. The finger shape estimation apparatus according to claim 5 . The nail region extracting unit uses pixel components of an image in which only a palm prepared in advance is captured using two principal axes among first to third principal axes set in advance by principal component analysis. The base plane is converted, the position of the separation plane along one of the two principal axes is lowered from a high position, and the area that has reached a high density area is defined as the separation plane, and the nail area is formed using the separation plane. The finger shape estimation device according to claim 5 or 6, wherein the finger shape estimation device extracts the shape. Acquiring an image including a finger shape;
Analyzing the acquired image to acquire a first feature amount corresponding to a ridge line shape of a finger included in the image;
A finger shape corresponding to the first feature value is obtained by referring to a database for collation in which a second feature value corresponding to a predetermined finger shape set in advance is stored based on the first feature value. Estimating,
Obtaining the first feature amount includes:
The hand image included in the image is the foreground, the image other than the finger image is the background, the distance from the background image in the foreground image is regarded as a height, the image is regarded as one mountain-shaped image, Including obtaining information on the ridge line shape from the mountain-shaped image.
Finger shape estimation method. Further, for the predetermined finger shape, including constructing the database in which at least angle data corresponding to the predetermined finger shape and the second feature amount are accumulated.
The finger shape estimation method according to claim 8 . Estimating the finger shape
Refining a data set from the database with a predetermined shape parameter in the finger shape, performing similarity calculation using the first feature amount for the narrowed data set group, and the similarity 10. The finger shape estimation method according to claim 8 , further comprising outputting a most similar data set based on a result of degree calculation . Obtaining the first feature amount includes:
Setting a starting point and an ending point of the ridge line vector based on the inclination of the ridge line vector obtained by contour scanning of the finger image included in the image,
Estimating the finger shape
The finger shape estimation method according to any one of claims 8 to 10, comprising estimating a finger shape based on a position of a start point and / or an end point of the set ridge line vector . Further, before estimating the finger shape, a separation plane is generated using only color information obtained from the image, nail region candidates are extracted based on the generated separation plane, and a preset palm is set. The finger shape estimation according to any one of claims 8 to 11, further comprising: extracting a nail region by re-determination of a nail region with respect to the nail region candidate by pixel discrimination using an image including an image. Method. Processing to acquire an image including a finger shape;
A process of analyzing the acquired image and acquiring a first feature amount corresponding to a ridge line shape of a finger included in the image;
A finger shape corresponding to the first feature value is obtained by referring to a database for collation in which a second feature value corresponding to a predetermined finger shape set in advance is stored based on the first feature value. The processing to be estimated is implemented in an information processing device and executed.
The process of acquiring the first feature amount includes:
The hand image included in the image is the foreground, the image other than the finger image is the background, the distance from the background image in the foreground image is regarded as a height, the image is regarded as one mountain-shaped image, Including processing for obtaining information on the ridge line shape from the mountain-shaped image
Finger shape estimation program. Further, before the process of estimating the finger shape, a separation plane is generated using only color information obtained from the image, nail region candidates are extracted based on the generated separation plane, and a preset palm is set. 14. The finger shape according to claim 13, wherein processing for extracting a nail region by performing redetermination of a nail region on the nail region candidate by pixel determination using an image including an image is implemented and executed in an information processing device. Estimation program.