JP6353660B2 - Sign language word classification information generation device and program thereof - Google Patents

Description

The present invention relates to a sign language word classification information generation apparatus and a program for generating a classification information for searching the sign language word.

In recent years, sign language interpreting and sign language broadcasting for visually impaired people have rapidly spread into society. Conventionally, a learning method using a sign language dictionary is generally used to learn the sign language.
The sign language dictionary includes a paper-based dictionary in which words and a series of images of sign language actions are associated with each other, and a recording medium such as a DVD in which words and videos of sign language actions are associated with each other.
In recent years, a technology has been disclosed in which sign language motion is acquired as motion of the human body including the fingertips (motion data) using a motion capture system, and the corresponding sign language is retrieved from the word by associating the word with motion data. (See Patent Document 1 and Non-Patent Document 1). According to this system, by specifying a word, it is possible to present CG animation of sign language motion data.
These methods (dictionaries, systems) search for sign language actions from words, not words from actual sign languages.

On the other hand, in order to search for a word from an actual sign language action, a dictionary for searching for a sign language from a hand pattern can be used (see Non-Patent Document 2). This dictionary associates large classifications centered on arm movements such as one-handed sign language, two-handed sign language, two-handed variant sign language, and small classifications centered on hand-types such as te-type and ho-type with words.
In addition, as a technique for searching for a word (character) from sign language action, from the image obtained by imaging the sign language action, by extracting the hand feature information by image processing, from the database that previously grouped the hand feature information in the sign language action, There is a technique for specifying characters corresponding to a group (see Patent Document 2).

JP 2011-112675 A JP 2012-252581 A

Naoto Kato, Hiroyuki Kaneko, Seiki Inoue, Toshihiro Shimizu, Yuji Nagashima: “Construction of a Japanese-Sign Language Bilingual Dictionary-Expansion of Japanese Vocabulary-”, HCG (Human Communication Group) Symposium, HCG2009-I-3 , (2009). Shigeru Takemura: “Sign Language / Japanese Dictionary”, Kosaido Publishing, April 1, 1994

In the method (dictionary, system) for searching for the sign language action from the above-mentioned word, the sign language action is associated with each word, and only the word is specified as a search condition. You can search.
On the other hand, there are many problems in the method of searching for words from actual sign language actions.
For example, in order to create a dictionary for searching for a word from a sign language action, the dictionary creator currently classifies sign language by looking at the sign language. Therefore, there is a problem that if the number of vocabularies in the dictionary is increased, the amount of work becomes enormous.
In addition, according to the technique for identifying a sign language word from a captured image by the above-described image processing, a hand and a finger may not be detected from the captured image depending on an environment such as illumination. In addition, the meaning of sign language varies depending on the shape (shape) of the hand. For this reason, it is very difficult to identify many three-dimensional hand types that differ by sign language words from a planar captured image.

The present invention has been made in view of such problems, the classification information for retrieving a word from the sign language operation, provide easily sign language word classification information generating apparatus and its program to produce without human intervention The task is to do.

  In order to solve the above problems, a sign language word classification information generation device according to the present invention generates classification information for search for searching for a sign language word from features of a hand shape, a position, and an orientation in the operation of a sign language word. A sign language word classification information generating apparatus, comprising a representative frame detection means, a hand type classification means, a hand position direction calculation means, a hand position classification means, and a hand direction classification means.

  In such a configuration, the sign language word classification information generation device uses the representative frame detection unit to generate motion in motion data including the relative positions of the joints of the human body and the amount of movement and rotation of the joints for each frame, which is a unit of the screen. The representative frame with the smallest amount is detected. This is because the movement of a manual word often rests once in the typical pose of that word.

Then, the sign language word classification information generation device clusters pairs of finger joint rotation amounts in the representative frame for each sign language word by the hand type classification means. As described above, by performing clustering according to the rotation amount of the finger joint, it is possible to classify the sign language words in rough hand shapes (types).
Then, the sign language word classification information generation apparatus associates the classification result as one piece of classification information for each motion data of the sign language word by the hand type classification means.
Thereby, the sign language word classification information generation device can generate classification information that can search for a sign language word from a hand pattern.

In addition, the sign language word classification information generating device uses the hand position direction calculating unit to calculate the wrist position based on the movement amount and the rotation amount of each joint from the joint serving as a predetermined reference position of the human body to the wrist joint for each representative frame. Calculate joint position and orientation.
Then, the sign language word classification information generation device classifies the wrist joint position by the position classified in advance by the hand position classification means, and the classification result is converted into the motion data of the sign language word corresponding to the representative frame. Correspond as one.
As a result, the sign language word classification information generation device can generate classification information capable of searching for a sign language word from the position of the hand.

The sign language word classification information generation device classifies the direction of the wrist joint by the direction classified in advance by the hand direction classification means, and classifies the classification result into motion data of the sign language word corresponding to the representative frame. Associate as one.
Thereby, the sign language word classification information generation device can generate classification information that can search for a sign language word from the direction of the hand (hand direction).

The present invention has the following excellent effects.
According to the present invention, sign language word motions defined by motion data can be classified according to the representative hand type, position, and orientation of the motion without human intervention.
According to the present invention, the sign language word indicated by the sign language action can be searched from the sign language action by the typical hand type, position, and orientation of the sign language word action.
Accordingly, the present invention can improve the learning efficiency of the sign language word motion.

It is a block block diagram which shows the structure of the sign language word classification | category information generation apparatus which concerns on embodiment of this invention. It is a data block diagram of the BVH format which shows an example of motion data. It is a figure which shows the joint node prescribed | regulated by motion data, Comprising: (a) is an example of the joint node of the whole body, (b) shows the example of the joint node of a right hand part. It is a figure for demonstrating the method of detecting a representative frame with the motion amount minimum frame detection means of FIG. 1, Comprising: It is a graph which shows the relationship between a frame and a motion amount. It is a figure which shows the phylogenetic tree produced | generated when the clustering means of FIG. 1 clustered using statistical processing software "R". It is explanatory drawing for demonstrating the area | region which classify | categorizes a hand position with the hand position classification | category means of FIG. It is a figure which shows the example of the hand type from which direction of a palm differs, Comprising: (a) is a figure which turned the back of the hand to the front, (b) is a figure which turned the palm to the front. It is a figure which shows the example from which the direction of an index finger differs, Comprising: (a) is a figure which turned the index finger inward, (b) is a figure which made the index finger face down. It is a flowchart which shows operation | movement of the sign language word classification | category information generation apparatus which concerns on embodiment of this invention. It is a block block diagram which shows the structure of the sign language word search apparatus which concerns on embodiment of this invention. It is a figure which shows the example of the search screen of a sign language word. It is a figure which shows the example of a screen which designates a hand type. It is a figure which shows the example of a screen which designates direction of a hand. It is a figure which shows the example of a screen which designates the direction of a finger | toe. It is a figure which shows the example of a screen which designates the position of a hand. It is a figure which shows the example of a screen which displays the candidate of the sign language word of a search result. It is a figure which shows the example of a screen which displays a sign language word operation | movement by selecting the sign language word of a search result. It is a flowchart which shows operation | movement of the sign language word retrieval apparatus which concerns on embodiment of this invention.

Embodiments of the present invention will be described with reference to the drawings.
[Outline of sign language word motion data]
First, the structure of motion data (sign language word motion data) indicating the operation of a sign language word used in the present invention will be described. This motion data is recorded for each sign language word using a general motion capture system. Here, as motion data, data in a file format (BVH [BioVision Hierarchy] format) developed by Biovision will be described as an example.

The motion data is composed of the relative position of the joint of the human body and the amount of movement and rotation of the joint for each frame, which is a unit of the screen when the sign language operation is reproduced.
FIG. 2 shows an example of motion data described in the BVH format.
As shown in FIG. 2, the motion data in BVH format is composed of a HIERARCHY section (from the first line to the 28th line) and a MOTION section (from the 29th line).
In the HIERARCHY part, a joint node (JOINT) indicating a joint of a human body is defined in a hierarchical structure, relative coordinates (OFFSET) from a parent joint node in an initial posture, and movement amounts (Xposition, Yposition, Zposition) of the joint node. And a variable of rotation amount (Zrotation, Xrotation, Yrotation).
In the MOTION section, the number of frames (line 30), the frame time interval (line 31), and the actual values (line 32) for the movement amount and rotation amount variables defined in the HIERARCHY section are stored in one frame. Each line is described one by one.

For example, as shown in FIG. 3, each joint node (JOINT) has a hierarchical structure in the order of connection with a predetermined reference position (here, Hips) as a root (ROOT). FIG. 3A shows an example of a joint node of a human body. FIG. 3B shows a finger joint node further connected to the right hand wrist (RightWrist) of FIG. In FIG. 3B, reference numerals of joint nodes other than the thumb and index finger are omitted.
For example, the position and orientation of the right hand joint node (RightIndex3) can be determined by following the hierarchy of joint nodes of the motion data shown in FIG. 2 as Hips, Chest1, Chest2, ..., RightWrist, RightIndex0, ..., RightIndex3. Can be sought.
Here, the XYZ coordinate system is a right-handed system as shown in FIG.
Hereinafter, after describing the configuration and operation of the sign language word classification information generation device according to the embodiment of the present invention, the configuration and operation of the sign language word search device will be described in order.

[Configuration of sign language word classification information generator]
First, the configuration of the sign language word classification information generation device 1 will be described with reference to FIG.
The sign language word classification information generation device 1 generates classification information that serves as a search condition for searching for a sign language word from the hand shape characteristics (hand type, position, orientation, etc.) in the operation of the sign language word.

This sign language word classification information generation device 1 uses hand data (hand type), palm direction (hand direction), hand finger (in this case, index finger) in the representative frame of the data from motion data indicating the operation of each sign language word. ) Direction (finger direction) and wrist position (hand position).
Here, the sign language word classification information generation device 1 includes representative frame detection means 10, type classification means 11, and position / direction classification means 12 in order to generate each classification information.

  The motion data D is the data described in FIG. 2 and is stored in the storage unit 2 in advance. The motion data D for each sign language word motion includes at least a file name, a unique number, etc. as management information. Is associated with a label L indicating the meaning of the sign language word (Japanese word).

  The representative frame detection means 10 detects, as a representative frame, a characteristic frame among the frames that change over time in the motion data D for each sign language word. The behavior of a sign language word often rests once in the typical pose of that word. Therefore, the representative frame detection unit 10 detects, as a representative frame, a frame having the smallest amount of motion between the start and end of the motion data D for each sign language word. Here, the representative frame detection unit 10 includes a motion amount calculation unit 100 and a motion amount minimum frame detection unit 101.

The motion amount calculation means 100 calculates a motion amount in units of frames in the motion data D for each sign language word.
Here, the motion amount calculation means 100 moves between the frame with the motion data D and the previous frame, the movement amount (X movement amount, Y movement amount, Z movement amount) and the rotation amount (X rotation amount, Y The amount of motion in a certain frame is obtained by calculating the sum of absolute values of differences for each component (rotation amount, Z rotation amount) or the square sum of the difference for each component. Similarly, the motion amount calculation unit 100 sequentially calculates the motion amount for all the frames.
That is, the motion amount calculation unit 100 calculates the motion amount between frames from the movement amount and the rotation amount of the frames described in units of one line from the 32nd line in the MOTION part of the motion data D described in FIG. Ask for.
Then, the motion amount calculation means 100 outputs the motion amount in frame units to the motion amount minimum frame detection means 101 for each sign language word motion.

The motion amount minimum frame detection unit 101 detects a frame having the minimum motion amount in the sign language word motion based on the motion amount calculated by the motion amount calculation unit 100.
Here, the motion amount minimum frame detecting means 101 detects a frame having the smallest motion amount between the frame in which the motion of the sign language word is started and the frame in which the motion of the sign language word is finished.

Here, with reference to FIG. 4, the frame detection process of the motion amount minimum frame detecting means 101 will be described.
FIG. 4 is a graph showing the amount of movement for each frame of a certain sign language word motion. The horizontal axis indicates the frame number, and the vertical axis indicates the amount of movement (no unit).
As shown in FIG. 4, the motion amount minimum frame detecting means 101 starts a frame that exceeds a predetermined threshold motion amount from the first frame of the sign language motion to the last frame, and starts the motion. And Further, as shown in FIG. 4, the motion amount minimum frame detection unit 101 ends the operation of a frame that exceeds a predetermined amount of motion from the last frame of the sign language operation toward the first frame. Frame Fe.
Then, the motion amount minimum frame detecting means 101 sets the frame having the minimum motion amount between the start frame Fs and the end frame Fe as the representative frame Ft.
The motion amount minimum frame detection unit 101 outputs information (identification information ID and frame number) for specifying the representative frame to the type classification unit 11 and the position / direction classification unit 12.
Returning to FIG. 1, the description of the configuration of the sign language word classification information generation device 1 will be continued.

The type classification unit 11 classifies the hand type in the representative frame for each of a plurality of sign language word motions detected by the representative frame detection unit 10 according to its shape.
Here, the type classification unit 11 includes a clustering unit 110 and a hand image generation unit 111.

The clustering means 110 clusters the hand molds of the representative frames of the plurality of motion data D.
Here, the clustering unit 110 uses a set of the rotation amounts of the joints of the finger portions in the representative frame of the motion data D as information for specifying the hand shape.
Specifically, the clustering unit 110 includes three joint nodes (Right Thumb 1 to Right Thumb 3) of the thumb, excluding the tip of each finger of the hand (here, right hand as an example) shown in FIG. 4 joint nodes (RightIndex0 to RightIndex3), 4 joint nodes of the middle finger, 4 joint nodes of the ring finger, and 4 joint nodes of the little finger, a set of rotation amounts of a total of 19 joint nodes. Clustering is performed on all detected representative frames as data units.
That is, the clustering unit 110 operates when all the representative frames of the motion data D stored in the storage unit 2 are detected from the representative frame detection unit 10.

A general method may be used for this clustering. For example, the clustering means 110 can use a K-average method.
When the K-average method is used, the clustering means 110 randomly assigns a set of rotation amounts of the finger joint nodes of the number of representative frames to K clusters. The clustering number K is, for example, a number that is predetermined as a sign language of sign language. Then, the clustering unit 110 obtains the center of gravity (the average of the X rotation amount, the average of the Y rotation amount, and the average of the Z rotation amount) for each cluster. Further, the clustering means 110 reassigns the K clusters to the clusters close to the center of gravity obtained previously. Then, the clustering unit 110 repeats the process until the cluster assignment does not change.
Thereby, the clustering means 110 can cluster the hand type in the representative frame into K pieces.

  Further, for example, the clustering means 110 may perform clustering using general statistical processing software “R”. When the statistical processing software “R” is used, the clustering means 110 displays the result of hand clustering on a display device that is not shown in the phylogenetic tree as shown in FIG. . Then, the operator inputs an appropriate number of clusters to the clustering means 110 based on the height in the system diagram. Then, the clustering unit 110 classifies each hand type divided in the system diagram according to the number of clusters.

Then, the clustering unit 110 sets the classified cluster-specific value as management information of the motion data D corresponding to the representative frame stored in the storage unit 2 as hand type information of the classification information C.
Further, the clustering unit 110 outputs information (identification information ID) for specifying the motion data D to the hand-type image generation unit 111 together with the cluster-specific value.

The hand-type image generation unit 111 generates an image (hand-type image) corresponding to the hand type classified by the clustering unit 110.
The hand image generation unit 111 generates a hand image G from the motion data D corresponding to the identification information ID input from the clustering unit 110 by CG (computer graphics).
The hand image generation unit 111 then writes and stores the generated hand image G in the storage unit 2 in association with the hand information that is a cluster-specific value input from the clustering unit 110. Note that the hand image generation unit 111 does not generate the hand image when the hand image G has already been generated for the hand information having the same value.
The hand-shaped image G generated in this way is used as a hand-shaped thumbnail image when searching for a sign language word.

The position / direction classification unit 12 classifies the positions and orientations of fingers in the representative frame for each of a plurality of sign language word motions detected by the representative frame detection unit 10.
Here, the position / direction classification unit 12 includes a hand position / direction calculation unit 120, a hand position classification unit 121, a hand direction classification unit 122, a finger direction calculation unit 123, and a finger direction classification unit 124.

The hand position / direction calculating unit 120 calculates the spatial position of the hand (wrist) and the direction of the hand (palm) represented by the representative frame.
This hand position / direction calculation means 120 calculates the movement amount and the rotation amount in the representative frame based on the hierarchical relationship (parent-child relationship) from the joint node at the reference position (Hips) described in the motion data D to the joint node of the wrist. Then, coordinate conversion is performed from the parent joint node toward the child joint node to obtain the wrist position and the palm direction.
Here, the hand position / direction calculation means 120 calculates the target wrist position and direction in the following order (procedure 1) to (procedure 3).

(Procedure 1)
For each joint node from the route (Hips) to the wrist (Wrist (here, referred to as “RightWrist”)), the hand position / direction calculating unit 120 calculates the movement amount and the rotation amount of the representative frame described in the motion data D, A transformation matrix (homogeneous transformation matrix) for obtaining the position and orientation of the joint node from the parent joint node is generated.
Specifically, the hand position / direction calculation unit 120 generates a transformation matrix A by performing a matrix operation represented by the following equation (1) for each joint node.
Here, the _{P x} movement amount in the X direction corresponding to XPosition, the moving amount of _{P y} is the Y direction corresponding to YPosition, _{P z} is the moving amount in the Z direction corresponding to ZPOSITION. Further, θ _x is a rotation amount around the X axis corresponding to Xrotation, θ _y is a rotation amount around the Y axis corresponding to Yrotation, and θ _z is a rotation amount around the Z axis corresponding to Zrotation.

(Procedure 2)
Then, the hand position / direction calculation unit 120 sequentially multiplies the conversion matrix corresponding to the route (Hips) calculated in (Procedure 1) to the conversion matrix corresponding to the wrist (Wrist), so that the entire route from the root to the wrist is obtained. Generate a transformation matrix of.
In other words, the transformation matrix Hips joints node _A Hips, the transformation matrix of Chest1 _A Chest1, ..., when the transformation matrix RightWrist was _{A RightWrist,} hand position direction calculating means 120, shown in the following equation (2) In this way, a transformation matrix B from the root to the wrist is generated by multiplying the respective matrices in order.

(Procedure 3)
Then, the hand position / direction calculating unit 120 calculates the wrist position and the palm direction from the components of the transformation matrix B calculated in (Procedure 2).
That is, the hand position direction calculation unit 120 calculates (LX, LY, LZ), which is a translation component of the transformation matrix B, so that the spatial coordinates (N _px , N _py , N _pz ) of the wrist position in the representative frame are calculated. )
The wrist position in the initial posture is a position obtained by adding relative coordinates (OFFSET) from the root (ROOT) to the wrist joint node in the motion data (sign language word motion data) D described with reference to FIG.

Further, the hand position / direction calculation means 120 is composed of the components (XX, XY, XZ, YX, YY, YZ, ZX, ZY, ZZ) of the transformation matrix B in the wrist direction vector in the initial posture of the sign language word motion. The direction vector (N _vx , N _vy , N _vz ) of the wrist that is the direction of the palm in the representative frame is obtained by multiplying the rotation matrix.
Note that the wrist direction vector in the initial posture is determined in advance as the initial pose of the sign language action. Here, the initial pose of the sign language operation is a posture in which both hands are spread sideways (T pose) and the palm is directed downward. That is, in the initial posture, the palm direction is downward, and the direction vector in the XYZ coordinate system is (0, -1, 0).
As described above, the hand position / direction calculation unit 120 can calculate the position and direction of the wrist in the space coordinates (XYZ coordinates) in the representative frame.
The hand position / direction calculating unit 120 outputs the calculated wrist position (spatial coordinates) to the hand position classifying unit 121 and outputs the wrist direction (direction vector) to the hand direction classifying unit 122.

The hand position classification unit 121 classifies the hand (wrist) position calculated by the hand position direction calculation unit 120.
The hand position classifying unit 121 classifies the position of the wrist in the representative frame according to a rough classification.
Here, as shown in FIG. 6, the hand position classifying unit 121 divides the representative frame image into three in the X direction (horizontal direction) and into three in the Y direction (vertical direction). , UC, UL, MR, MC, ML, DR, DC, DL), it is determined where the wrist is located.

That is, the hand position classification unit 121 uses the X coordinate N _px , Y coordinate N _py of the spatial coordinates (N _px , N _py , N _pz ) of the wrist position calculated by the hand position direction calculation unit 120 to determine the hand ( Classify the position of the wrist.
Here, the hand position classifying unit 121 classifies the position on the XY plane. However, the hand position classifying unit 121 may classify the position on the three-dimensional space by adding the Z coordinate of the wrist position.
Then, the hand position classification unit 121 uses the unique value that identifies the divided region where the wrist is located as the hand position information of the classification information C, and the sign language word (motion data D) identified by the identification information ID in the storage unit 2. ) To the management information corresponding to.

The hand direction classifying unit 122 classifies the direction of the hand (wrist) calculated by the hand position / direction calculating unit 120.
The hand direction classifying means 122 classifies the direction of the palm direction in the representative frame according to a rough classification.
Here, among the direction vectors (N _vx , N _vy , N _vz ) calculated by the hand position / direction calculating means 120, the axis component having the largest absolute value is classified as the palm direction together with its sign. To do.

For example, in FIG. 7A, the sign language has the back of the hand facing forward, and in FIG. 7B, the palm is facing the front. In this case, among the wrist direction vectors (N _vx , N _vy , N _vz ), even if the absolute value of the Z-axis component Nvz is the largest and the hand shape is the same, FIG. ) Can be classified according to the sign of the Z-axis component.
Note that the hand direction classification unit 122 does not necessarily need to classify the directions in all of the XYZ directions. For example, focusing on only the Z-axis direction, classification may be performed based on the positive / negative.
Then, the hand direction classifying means 122 corresponds to the sign language word (motion data D) specified by the identification information ID in the storage means 2 with the unique value specifying the palm direction as the hand direction information of the classification information C. Set the management information to be used.

The finger direction calculation means 123 calculates the direction of the finger of the hand represented by the representative frame.
The finger direction calculation means 123 calculates two predetermined joint positions of the finger from the motion data D of the representative frame, and calculates the direction indicated by the finger from the relative position of each joint.

For example, when obtaining the direction of the index finger of the right hand, the finger direction calculation means 123, in the joint node shown in FIG. 3, the position of the joint node (RightIndexEff) at the tip of the index finger and the joint node (RightIndex1) at the base of the finger Find the position of.
The method for obtaining the positions of these joint nodes is the same as the method for obtaining the wrist joint node (RightWrist) by the hand position direction calculating means 120.
However, since the movement amount and the rotation amount from the joint node of the reference position (Hips) to the wrist joint node (RightWrist) have already been calculated by the hand position direction calculation unit 120, the finger direction calculation unit 123 Using the amount of movement and the amount of rotation, calculation is performed for the subsequent joint nodes of the finger, and the respective spatial coordinates may be obtained.

Then, the finger direction calculation unit 123 uses the spatial coordinates (I1x, I1y) indicating the position of the root joint node (RightIndex1) from the spatial coordinates (Iex, Iey, Iez) indicating the position of the joint node (RightIndexEff) at the tip of the index finger. , I1z) is calculated. The difference between the components becomes an index (direction vector) indicating the direction of the finger on each axis.
Then, the finger direction calculation unit 123 outputs the finger direction vector to the finger direction classification unit 124.

The finger direction classifying unit 124 classifies the finger direction calculated by the finger direction calculating unit 123.
This finger direction classifying means 124 is the positive direction of the axis if the sign is positive in the axial direction having the largest absolute value among the components of the finger direction vector calculated by the finger direction calculating means 123. If the sign is negative, the finger is classified in the negative direction of the axis.

For example, in FIG. 8A, the signer points the index finger inward (X-axis positive direction), and in FIG. 8B, the index finger is directed downward (Y-axis negative direction). Thus, even if the hand type is the same, FIGS. 8A and 8B can be classified into different sign language words depending on the direction of the index finger.
Note that the finger direction classification unit 124 does not need to classify directions for all fingers, and may classify only the three fingers, for example, the index finger only, the thumb, the index finger, and the middle finger.
Then, the finger direction classifying unit 124 uses a unique value for specifying the finger direction as finger direction information of the classification information C and corresponds to the sign language word (motion data D) specified by the identification information ID in the storage unit 2. Set the management information to be used.

By configuring the sign language word classification information generation device 1 as described above, the sign language word classification information generation device 1 generates the classification information based on the hand type and the finger finger direction from the motion data indicating the sign language word motion. Can do.
By using this classification information as a search condition, it becomes possible to search for a sign language word indicated by a sign language action from the hand shape and the position and orientation of a finger.
The sign language word classification information generation apparatus 1 can operate a computer with a sign language word classification information generation program for causing a computer to function as each of the means described above.

[Operation of sign language word classification information generator]
Next, the operation of the sign language word classification information generation device 1 will be described with reference to FIG.
First, the sign language word classification information generation apparatus 1 detects a representative frame from the motion data D for each sign language word stored in the storage unit 2 in the representative frame detection unit 10.
That is, the sign language word classification information generation device 1 uses the motion amount calculation unit 100 of the representative frame detection unit 10 to calculate the difference between the movement amount and rotation amount components between the frame having the motion data D and the previous frame. The amount of motion in units of frames is calculated by calculating the sum of absolute values or the sum of squares of differences for each component (step S1).

  Then, the sign language word classification information generating device 1 includes a frame in which the movement of the sign language word is started and a frame in which the movement of the sign language word is ended by the minimum motion amount frame detection unit 101 of the representative frame detection unit 10. The frame with the smallest amount of motion calculated in step S1 is detected as a representative frame (step S2).

  Here, when the representative frame detection unit 10 has not detected a representative frame for all the motion data D stored in the storage unit 2 (No in step S3), the process returns to step S1 to detect the representative frame. repeat. On the other hand, when the detection of the representative frame is completed for all the motion data D (Yes in step S3), the representative frame detection means 10 determines the identification information ID of the motion data D and the frame number of the representative frame as the type classification means 11. The position / direction classification means 12 is notified, and control is transferred to the respective classification means 11 and 12 (not shown as steps).

After that, the sign language word classification information generating device 1 uses the clustering unit 110 of the type classification unit 11 to set a pair of rotation amounts of the joints of the finger portions in the representative frame as one data unit, and perform hand type clustering for all the representative frames. Perform (step S4).
Then, the sign language word classification information generation device 1 stores the cluster-specific values clustered in step S4 by the clustering unit 110 in association with the motion data D corresponding to the representative frame as the hand pattern information of the classification information C. (Step S5).

Further, the sign language word classification information generation device 1 generates an image (hand image) corresponding to the hand clustered in step S4 by the hand image generation unit 111 of the type classification unit 11 (step S6).
Then, the sign language word classification information generation device 1 stores the hand image generated in step S6 in association with the hand image information by the hand image generation unit 111 (step S7).

In addition, the sign language word classification information generation device 1 classifies the position of the hand and the direction of the finger in each representative frame by the position / direction classification unit 12.
That is, the sign language word classification information generating device 1 performs coordinate conversion based on the hierarchical relationship of the joint nodes described in the motion data D by the hand position / direction calculation unit 120 of the position / direction classification unit 12, thereby ) And the direction of the hand (palm) are calculated (step S8).
Then, the sign language word classification information generating device 1 classifies the position of the hand (wrist) calculated in step S8 by the hand position classification unit 121 of the position / direction classification unit 12 in advance, and classifies the classified position. The specified value is stored as hand position information of the classification information C in association with the motion data D corresponding to the representative frame (step S9).

  Further, the sign language word classification information generation device 1 classifies the direction of the hand (palm) calculated in step S8 by the hand direction classification unit 122 of the position / direction classification unit 12 in advance, and determines the classified direction. The specified value is stored as hand direction information of the classification information C in association with the motion data D corresponding to the representative frame (step S10).

Furthermore, the sign language word classification information generation device 1 performs finger-pointing (index finger) by performing coordinate transformation based on the hierarchical relationship of the joint nodes described in the motion data D by the finger direction calculation unit 123 of the position / direction classification unit 12. Is calculated (step S11).
Then, the sign language word classification information generation device 1 classifies the finger directions calculated in step S11 by the finger direction classification unit 124 of the position / direction classification unit 12 according to a previously classified direction, and specifies the classified direction. Is stored in association with the motion data D corresponding to the representative frame as the finger direction information of the classification information C (step S12).

Here, if the classification of the hand position and the finger direction is not completed for all the representative frames (No in step S13), the position / direction classification unit 12 returns to step S8 and repeats the classification process. .
When the classification of the hand position and the finger direction is completed for all the representative frames (Yes in step S13), the sign language word classification information generation device 1 ends the operation.

With the above operation, the sign language word classification information generation device 1 performs the classification information C (hand type information, hand position information, hand direction information, finger direction) for the motion data D stored in the storage unit 2. Information).
Further, the sign language word classification information generation device 1 can generate a hand image as a thumbnail image for search in association with the hand information.

[Configuration of sign language word search device]
Next, the configuration of the sign language word search device 3 will be described with reference to FIG.
The sign language word search device 3 searches for a sign language word using the classification information generated by the sign language word classification information generation device 1 (see FIG. 1) as a search condition. Note that the sign language word search device 3 is connected to the motion data D with the storage means 2 in which the classification information and the like are set in advance by the sign language word classification information generation device 1 (see FIG. 1).
Here, the sign language word search device 3 includes a search screen display control means 30, a classification information designation means 31, and a search execution means 32.

The search screen display control unit 30 displays a screen for inputting a search condition for sign language words and controls a search screen serving as a user interface for receiving the search condition from the user.
This search screen display control means 30 is a sign language word for the user to specify the classification information C (hand type information, hand position information, hand direction information, finger direction information) stored in the storage means 2 as a search condition. A search screen is generated and displayed on the screen of the display device M.
For example, as shown in FIG. 11, the search screen display control means 30 specifies the title part 51 of the pull-down menu for specifying the hand shape as the search condition and the direction of the hand (the direction of the palm) as the search condition. A search screen 50 including the title part 52 of the pull-down menu and the title part 53 of the pull-down menu for specifying the direction of the finger as a search condition is displayed as a button for selecting an item of the search condition.

Further, as shown in FIG. 11, the search screen display control means 30 displays an area designation image 54 including a check area (radio button 54a) for designating the hand position on the screen.
Further, as shown in FIG. 11, the search screen display control means 30 displays a search button 55 on the screen for accepting execution of the search after specifying the search conditions.

This search screen display control means 30 transfers control to the hand type designation means 310 of the classification information designation means 31 described later by selecting the title portion 51 with a selection means such as a mouse.
Further, the search screen display control means 30 transfers the control to the hand direction designation means 311 of the classification information designation means 31 to be described later by selecting the title portion 52 with the selection means such as a mouse.
Further, the search screen display control means 30 transfers the control to the finger direction designation means 312 of the classification information designation means 31 described later by selecting the title portion 53 with a selection means such as a mouse.
Further, the search screen display control means 30 transfers the control to the hand position designation means 313 of the classification information designation means 31 to be described later by selecting the region designation image 54 with a selection means such as a mouse.

The classification information designating unit 31 receives designation of search conditions from the user via the search screen.
Here, the classification information specifying unit 31 includes a hand type specifying unit 310, a hand direction specifying unit 311, a finger direction specifying unit 312, and a hand position specifying unit 313.

The hand type designation unit 310 receives a hand type as a search condition.
The hand type designation unit 310 displays the hand type image G stored in the storage unit 2 as a thumbnail image when the hand type is designated as a search condition on the search screen from the search screen display control unit 30. . And the hand type designation | designated means 310 specifies hand type information corresponding to it as a search condition by selecting a thumbnail image with selection means, such as a mouse | mouth.

For example, as shown in FIG. 12, when the title portion 51 is selected on the search screen 50, the hand type designation unit 310 displays thumbnail images 51a indicating a plurality of hand types and accepts the selection.
This hand type designation means 310 moves the control to the search screen display control means 30 after the hand type is selected and the hand type information is specified as one of the search conditions.
Note that the hand type information specified as the search condition is held in a memory (not shown) and is referred to by the search execution means 32 when executing the search.

The hand direction designation means 311 accepts the direction of the hand (palm) as a search condition.
This hand direction designating means 311 indicates the hand direction information stored in the storage means 2 when the hand direction (the palm direction) is designated as a search condition on the search screen from the search screen display control means 30. Display category items. And hand direction designation | designated means 311 specifies the hand direction information corresponding to it as a search condition by selecting an item with selection means, such as a mouse | mouth.

For example, as shown in FIG. 13, when the title portion 52 is selected on the search screen 50, the hand direction designating unit 311 displays the item content that has been classified in advance as the hand direction and accepts the selection. .
The hand direction designating unit 311 moves the control to the search screen display control unit 30 after the hand direction is selected and the hand direction information is specified as one of the search conditions.
Note that the hand direction information designated as the search condition is held in a memory (not shown) and is referred to by the search execution means 32 when executing the search.

The finger direction designating unit 312 accepts the finger direction as a search condition.
The finger direction designating unit 312 stores the finger direction information stored in the storage unit 2 when the search screen display control unit 30 designates the finger direction (direction pointed by the index finger) as a search condition on the search screen. Displays the category items. The finger direction designating unit 312 selects the item with a selection unit such as a mouse, and specifies the corresponding finger direction information as a search condition.

For example, as shown in FIG. 14, when the title portion 53 is selected on the search screen 50, the finger direction designating unit 312 displays the item content that has been classified in advance as the finger orientation and accepts the selection. .
The finger direction designating unit 312 transfers the control to the search screen display control unit 30 after the finger direction is selected and the finger direction information is specified as one of the search conditions.
Note that the finger direction information designated as the search condition is held in a memory (not shown) and is referred to by the search execution means 32 when executing the search.

The hand position designation means 313 accepts the hand position as a search condition.
The hand position designation means 313 specifies hand position information corresponding to the hand position information as a search condition by designating the hand position on the search screen from the search screen display control means 30.
For example, as shown in FIG. 15, by specifying the area of the area specifying image 54 on the search screen 50, the hand position specifying means 313 changes the color or the like of the radio button 54a to select the area. Represents that.
This hand position designation means 313 moves the control to the search screen display control means 30 after the hand position is selected and the hand position information is specified as one of the search conditions.
Note that the hand position information designated as the search condition is held in a memory (not shown) and is referred to by the search execution means 32 when executing the search.

The search execution means 32 receives a search execution instruction from the user via the search screen.
Here, the search execution means 32 includes search means 320, search result display means 321, and sign language action display means 322.

The search unit 320 searches for a sign language word that matches the search condition specified by the classification information specifying unit 31.
This search means 320 receives an instruction from the search screen display control means 30 for executing a search on the search screen (for example, when the search button 55 is pressed on the search screen 50 in FIG. 11). The label L of the sign language word having the classification information C that matches the search conditions (hand type information, hand position information, hand direction information, finger direction information) held by the information specifying means 31 is searched.
The search means 320 does not need to specify all the search conditions by the classification information specifying means 31, but searches for the label L of the sign language word having the classification information C that matches only the specified search conditions.
Then, the search unit 320 outputs the label L that is the search result to the search result display unit 321.

The search result display unit 321 displays a sign language word (label) that is a result of the search performed by the search unit 320.
For example, as shown in FIG. 16, the search result display unit 321 displays the sign language word of the search result in the search result word area 60. When there are a plurality of sign language words as a search result, the plurality of sign language words are listed and displayed.

The sign language action display means 322 displays the action of the sign language word displayed by the search result display means 321 in CG.
The sign language action display unit 322 reads the motion data D corresponding to the selected sign language word and reproduces it as a CG moving image by selecting the sign language word displayed by the search result display unit 321. As a method for generating CG from the motion data D, a general method may be used.

For example, as shown in FIG. 17, when a plurality of sign language word candidates (words A to C) are displayed in the search result word area 60 as a search result, and one of them (word A) is selected, sign language is displayed. The action display means 322 reads out the motion data D with the word A as the label L from the storage means 2 and reproduces the sign language word action in the CG drawing area 62 by CG.
As a result, for example, even when there are a plurality of sign language words that match the search condition, the user can search for a sign language word corresponding to a desired sign language operation by confirming the sign language operation.

By configuring the sign language word search device 3 as described above, the sign language word search device 3 can search for the sign language word indicated by the sign language action from the hand type, the position and direction of the finger during the sign language word action. it can.
The sign language word search device 3 can be operated by a sign language word search program for causing a computer to function as each of the means described above.

[Operation of sign language word classification information generator]
Next, the operation of the sign language word search device 3 will be described with reference to FIG.
First, the sign language word search device 3 causes the search screen display control means 30 to display a sign language word search screen 50 as shown in FIG. 11 on the display device M (step S20).
Then, the sign language word search device 3 branches the processing operation according to the selection position on the search screen 50 selected by the search screen display control means 30 through the selection means such as a mouse from the user (step S21). .

  When “hand type” is selected as the search condition item on the search screen 50, the sign language word search device 3 uses the hand type designation unit 310 of the classification information designation unit 31 to store the hand type stored in the storage unit 2. The mold image G is displayed as a thumbnail image (see FIG. 12; step S22). The hand type designation unit 310 then transfers control to the search screen display control unit 30.

  When one of the handprints (thumbnail images) displayed in step S22 is selected on the search screen 50 as the content of the handprint, the sign language word search device 3 specifies the handprint designation of the classification information designating unit 31. The hand type information corresponding to the selected hand type is held as a search condition by means 310 (step S23). The hand type designation unit 310 then transfers control to the search screen display control unit 30.

  When “hand direction” is selected as the search condition item on the search screen 50, the sign language word search device 3 uses the hand direction specification means 311 of the classification information specification means 31 to classify the classification items of the hand direction information. A candidate for the palm orientation to be displayed is displayed (see FIG. 13; step S24). Then, the hand direction designating unit 311 transfers control to the search screen display control unit 30.

  When one of the hand direction candidates displayed in step S24 is selected on the search screen 50, the sign language word search device 3 uses the hand direction specifying unit 311 of the classification information specifying unit 31 to select the selected hand. Hand direction information corresponding to the direction is held as a search condition (step S25). Then, the hand direction designating unit 311 transfers control to the search screen display control unit 30.

  In addition, when “finger direction” is selected as the search condition item on the search screen 50, the sign language word search device 3 uses the finger direction specification unit 312 of the classification information specification unit 31 to identify the classification item of the finger direction information. The finger direction candidates are displayed (see FIG. 14; step S26). Then, the finger direction specifying unit 312 transfers control to the search screen display control unit 30.

  When one of the finger direction candidates displayed in step S26 is selected on the search screen 50, the sign language word search device 3 uses the finger direction specifying unit 312 of the classification information specifying unit 31 to select the selected finger direction. The finger direction information corresponding to the direction is held as a search condition (step S27). Then, the finger direction specifying unit 312 transfers control to the search screen display control unit 30.

  When the “hand position” area is selected as the search condition on the search screen 50, the sign language word search device 3 displays the selected area by the hand position specifying means 313 of the classification information specifying means 31. (See FIG. 15) The hand position information corresponding to the selected hand position is held as a search condition (step S28). Then, the hand position designation unit 313 transfers control to the search screen display control unit 30.

  When the search button for instructing execution of the search is pressed on the search screen 50, the sign language word search device 3 uses the search means 320 of the search execution means 32 to perform steps S 23, S 25, and S 27 as necessary. The sign language word that matches the search condition specified in step S28 is searched, and the search result display means 321 displays the sign language word of the search result (see FIG. 16; step S29).

Further, when the sign language word candidate displayed in step S29 is selected, the sign language word search device 3 reads the motion data D corresponding to the selected sign language word by the sign language action display means 322 of the search execution means 32. Then, the sign language word motion is reproduced as a CG video (step S30).
With the above operation, the sign language word search device 3 can search for the sign language word indicated by the sign language action from the hand type, the position and the direction of the fingers during the sign language word action selected by the user.

[Modification]
The configuration and operation of the sign language word classification information generation device and the sign language word search device according to the embodiment of the present invention have been described above, but the present invention is not limited to this embodiment.
For example, here, the sign language word classification information generation device generates classification information based on the right hand type, position, orientation, and right finger direction in order to make the explanation easy to understand.
However, the sign language word classification information generation device may generate classification information further according to the left hand type, position, orientation, and left finger direction, as with the right hand.
In that case, the sign language word search apparatus may similarly search the sign language word by adding the left hand classification information as a search condition.

Further, here, the sign language word classification information generation device generates a hand image when classifying a hand shape, but it is not always necessary to generate a hand image.
In that case, it is good also as matching a known name (for example, te type | mold, e-type etc.) for every hand type | mold classified beforehand.

Here, the sign language word classification information generation device generates classification information that classifies the hand type, position, orientation, and finger direction.
However, the sign language word classification information generation device does not necessarily need to generate classification information for all these items. For example, the classification information that classifies the direction of the finger may be omitted, and a simple configuration may be used.

Here, the sign language word classification information generation device specifies the position of the hand by the position of the wrist. However, for example, the position may be specified by a joint node at the tip of the index finger.
Also, here, the sign language word retrieval apparatus handles the hand type, position, orientation, and finger direction independently.
However, the sign language word search apparatus may search for a sign language word by specifying, for example, the left and right hand types are the same and the left and right hand positions and directions are symmetrical as search conditions.

DESCRIPTION OF SYMBOLS 1 Sign language word classification | category information generation apparatus 10 Representative frame detection means 100 Motion amount calculation means 101 Motion amount minimum frame detection means 11 Type classification means 110 Clustering means 111 Hand type image generation means 12 Position / direction classification means 120 Hand position / direction calculation means 121 Hand Position classification means 122 Hand direction classification means 123 Finger direction calculation means 124 Finger direction classification means 2 Storage means 3 Sign language word search device 30 Search screen display control means 31 Classification information designation means 310 Hand type designation means 311 Hand direction designation means 312 Finger direction Designation means 313 Hand position designation means 32 Search execution means 320 Search means 321 Search result display means 322 Sign language action display means C Classification information D Motion data G Hand image

Claims (4)

A sign language word classification information generation device that generates classification information for search for searching for the sign language word from features of a hand shape, a position, and an orientation in the operation of the sign language word,
In motion data composed of the relative position of the joints of the human body and the amount of movement and rotation of the joint for each frame, which is a unit of the screen, the representative amount of movement is minimized between the start and end of the operation. Representative frame detecting means for detecting a frame for each sign language word;
The sign language words are classified by hand shape by clustering the set of rotation amounts of the joints of the fingers in the representative frame for each sign language word detected by the representative frame detection means, and the classification result of the sign language words Hand type classification means for associating each piece of motion data as one of the classification information;
For each representative frame, hand position / direction calculation for calculating the position and orientation of the wrist joint based on the amount of movement and the amount of rotation of each joint from the joint serving as a predetermined reference position of the human body to the joint of the wrist Means,
The position of the wrist joint calculated by the hand position / direction calculating means is classified by a pre-divided position, and the classification result is used as one of the classification information in the motion data of the sign language word corresponding to the representative frame. Hand position classification means to associate;
The wrist joint direction calculated by the hand position / direction calculating means is classified according to the direction divided in advance, and the classification result is used as one of the classification information in the motion data of the sign language word corresponding to the representative frame. Hand direction classification means to associate;
A sign language word classification information generating device comprising: Finger direction calculation means for obtaining two predetermined joint positions of the finger based on the amount of movement and the amount of rotation for each representative frame, and calculating a direction indicated by the finger from the relative position of each joint;
The direction indicated by the finger calculated by the finger direction calculation means is classified according to a previously classified direction, and the classification result is associated with motion data of a sign language word corresponding to the representative frame as one of the classification information. Classification means;
The sign language word classification information generating device according to claim 1, further comprising:   For each hand type classified by the hand type classification means, a hand type image that becomes a thumbnail image for search is generated by CG from one of the representative frames of sign language words classified into the hand type. The sign language word classification information generation device according to claim 1, further comprising a type image generation unit.   A sign language word classification information generation program for causing a computer to function as each means of the sign language word classification information generation device according to any one of claims 1 to 3.