JPH08115408A - Finger language recognition device - Google Patents

Abstract

PURPOSE: To flexibly, efficiently and precisely execute a finger language recognition by recognizing a manual operation pattern in an operation element unit at first, to synthesize the recognized result of an operation element so as to recognize a word and recognizing a finger language word including an operation changing in an expressed context and a situation by means of laying emphasis on a constitution element which does not change. CONSTITUTION: A finger language input part 101 converts the operation in finger language into an electric signal. An operation element recognition part 102 recognizes the constitution element of the operation constituting the finger word from operation data. Recognized result output recognized in the operation element recognition part 102 is stored in an operation element recognized result storage part 110. A finger language word recognition part 111 collates information on the operation elements stored in the operation element recognized result storage part 110 with that of a finger language word dictionary 112, and recognizes the finger language word according to whether they are matched with each other or not. Information on the finger language words executing recognition are stored in the finger language word dictionary 112 as the simultaneous and successive combination of marks showing the operation elements constituting the finger language words.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a sign language recognition apparatus for recognizing sign language, converting the sign language into a speech language and outputting the same, and in particular, by inputting sign language and outputting the result in the form of a speech language. The present invention relates to a sign language recognition device capable of supporting communication between a deaf person and a hearing person.

[0002]

2. Description of the Related Art As a conventional sign language recognition device, a device using a glove (data glove) is used to input hand motion data of sign language, and general pattern matching such as voice recognition or image recognition is performed on the data. There was a method of recognizing sign language by using the technology or the technology of the neural network. Further, in the technology of voice recognition and image recognition, which is a field close to sign language recognition, the feature amount necessary for recognizing a certain pattern is calculated, and the result is used for recognition. In this case, there is a method of directly recognizing an object from some characteristic parameters, but as another method, a method of first recognizing elements constituting a large structure such as a phoneme or a straight line and then recognizing the entire object by the combination thereof. There is. In addition, Japanese Patent Application No.
In the specification and drawings (Sign Language Interpreter), a sign gesture or a facial expression is inserted in the boundary of the sign language words, or a signal is input from another input device at the boundary of the sign language words to detect the boundaries of the sign language words. Is made clear, and sign language sentences are recognized by the recognition of each sign language word. However, even in this method, the sign language words may change depending on the context or the surrounding situation, so that the boundaries of the sign language words may not be clearly recognized. Furthermore, Japanese Patent Application No. 5-12569 proposed by the applicant prior to the present application
In No. 8 and the drawings, the recognition of hand movement was word recognition. This causes a problem that the sign language word changes depending on the context and surrounding conditions, as described above.

[0003]

Generally, in sign language operation, sign language sentences are expressed by continuously connecting units called sign language words, and conversation is performed by exchanging sign language sentences. . In this case, the sign language words include a sign language word that always performs the same operation and a sign language word that partly changes depending on the context and surrounding circumstances. Therefore, in order to recognize general sign language, it is necessary to clearly separate the action of the sign language word from the part that does not change depending on the context or the situation. However, in the conventional sign language recognition technology, the entire input sign language data is
In most cases, recognition was performed by considering it as one pattern, which made it impossible to recognize general sign language that changes depending on the context or situation. On the other hand, in the recognition of an artificially created sign language word, there is also a technique in which the motion of the sign language word is separated into several elements and a change in a specific element is given a specific meaning for recognition. For example, in sign language showing the ending of an English word stem, in order to express the past, past completion, etc. in sign language, if the hand is downward even if the same shape, the past if the hand is turned sideways. There is a method to decide it as completed. However, in general sign language, the part that changes depending on the context or the situation is not common and differs depending on the type of sign language word, so such sign language cannot be dealt with. Further, since the recognition of the stationary state is the basic, it is not possible to recognize sign language in which the way the shape of the hand changes or the way the hand moves is meaningful.

In the end, in order to recognize such sign language, the action is decomposed into its constituent elements, and after recognizing each constituent element, the combination of the recognition results of the constituent elements is focused on the constituent element with little change. I think it is better to make a recognition by investigating in advance. However, in the case of sign language, there is a problem that the constituent elements of the operation are combined not only sequentially but also simultaneously. In voice recognition and image recognition, there is a method of recognizing a target component and recognizing the entire target from the combination, but if limited to voice recognition among them, it is only a sequential combination of the components, and As far as image recognition is concerned, basically only simultaneous combinations are handled. Therefore, when a plurality of elements are combined simultaneously and sequentially like sign language, it cannot be handled by the processing method of image recognition or voice recognition. An object of the present invention is to solve such a conventional problem and to flexibly, efficiently, and accurately recognize a general sign language having a part that changes depending on the expressed context or situation. To provide a recognition device.

[0005]

In order to achieve the above object, the sign language recognition apparatus of the present invention decomposes a motion in a sign language word into motion elements such as a hand shape, direction, position, way of movement and relation, and A sign language word is expressed as a combination of symbols that represent the action elements of. Some of those combinations are
The operation elements connected sequentially (in time series) and the operation elements connected simultaneously (in parallel) are included. Then, in the recognition process, first, the motion elements of the necessary motions (sequential elements and simultaneous elements) are individually recognized and stored in the storage means. Next, from the recognition result stored in the storage means, a motion element required for the sign language word to be recognized is searched for, and the sign language word is searched by examining the simultaneous and sequential combinations of the searched motion components. recognize. In addition, for the recognition of the constituent elements of each operation, a recognition method optimal for the recognition of each constituent element is used.

[0006]

In the present invention, a sign language word is expressed as a combination of motion elements, and the simultaneous and sequential combinations of recognition results of motion elements that have been independently recognized are evaluated,
Recognize sign language words. That is, in the conventional sign language recognition device, the overall hand movement pattern is compared with the standard hand movement pattern stored in the word dictionary and is recognized depending on whether or not they match. However, in the present invention, the hand movement pattern is first recognized in units of movement elements (for example, refer to the partial patterns appearing during the sign language movement of FIGS. 1 and 8). At this stage, it has nothing to do with the particular word. Then, the results of the action elements are integrated to recognize the word. Therefore,
Since it is recognized in smaller units (motion elements), processing is easy, and even when recognizing words, it is only necessary to integrate information regarding the presence or absence of motion elements, so processing is easy and high-speed processing is possible. Is. As a result, it is possible to recognize the sign language words by focusing on the components that do not change, even with respect to the sign language words that include actions that change depending on the context or situation in which they are expressed. Furthermore, since each motion element can be recognized using a method suitable for the property of each motion element, it is possible to perform sign language recognition that is flexible, efficient, and accurate.

[0007]

Embodiments of the present invention will now be described in detail with reference to the drawings. FIG. 1 is a conceptual block diagram of a sign language recognition apparatus showing an embodiment of the present invention. In FIG. 1, a sign language input unit 101 converts a motion in sign language into an electric signal. The motion element recognizing unit 102 recognizes a motion constituent element (hereinafter referred to as a motion element) that constitutes a sign language word from the motion data. The motion element recognition unit 102 is composed of independent recognition units 103, 105, and 107 for respective motion elements. Each of the motion element recognition units 103, 105 and 107 has a recognition parameter 10 required for the recognition process.
4, 106 and 108 are prepared. Motion element recognition unit 10
In 2, the recognition of each motion element is performed independently for the same input data. When the recognition is completed by one motion element recognition unit, it can be seen that the motion element corresponding to the recognition unit was included. The recognition result output recognized by the motion element recognition unit 102 is stored in the motion element recognition result storage unit 110. Motion element recognition result storage unit 1
At 10, information on the recognized motion elements is stored. The next sign language word recognition unit 111 collates the information of the motion elements stored in the motion element recognition result storage unit 110 and the sign language word dictionary 112, and recognizes the sign language words based on whether they match. In the sign language word dictionary 112, information on the sign language words to be recognized is stored as a simultaneous and sequential combination of symbols representing the action elements forming the sign language words. Finally, the sign language word recognized by the sign language word recognition unit 111 is output to the outside via the output unit 113.

FIG. 2 is a block diagram showing an example of the hardware for realizing the sign language recognition apparatus shown in FIG. In FIG. 2, a hand movement input device 201 is a device for converting hand movement in sign language into an electric signal, and a sensor is installed on a glove,
A device (for example, a data glove) that is well known as a device that converts the shape or motion of a hand into an electric signal can be used. The hand movement input device 201 converts the hand movement of sign language into multidimensional time-series data including the bending angle of the finger, the position of the hand, and the like. That is, the data is converted into dimensional data classified by the position, direction, and bending of the finger as shown in FIG. 3 described later, and in the case of FIG. 3, converted into 16-dimensional data. The arithmetic device 202 is a device for recognizing motion elements and sign language words, and includes the memory 20.
A program is read from 4, 205 and recognition processing is performed according to the program. The output device 203 is a device that outputs the recognition result of the sign language word, and can use the output by characters or the output by voice using voice synthesis. The memory 204 is a storage device for storing a program for recognizing an operating element, and is a memory 205.
Is a storage device for storing a program for recognizing a sign language word. The memory 206 is a storage device for storing parameters necessary for recognizing each action element, and the memory 207 is a sign language word representing a combination of action elements necessary for recognizing each sign language word. A memory 208 is a storage device for storing a dictionary.
Is a storage device for storing a recognition result of a motion element for reference in sign language word recognition.

FIG. 3 is a format diagram of the hand movement data converted by the sign language input device of the present invention. In FIG. 3, reference numeral 301 denotes data relating to the position of the hand.
It is composed of z-axis data 304. Reference numeral 305 is data regarding the direction of the hand, and the hand direction further includes an angle 306 around the x axis, an angle 307 around the y axis, and an angle 308 around the z axis. Reference numeral 309 denotes data relating to bending of the finger. The bending of the finger further includes bending angle 310 of the second joint of the thumb, bending angle 311 of the third joint of the thumb, bending angle 312 of the first joint of the index finger, and second joint of the index finger. Bending angle 313, bending angle 314 of first joint of middle finger, bending angle 315 of second joint of middle finger, bending angle 316 of first joint of ring finger, bending angle 317 of second joint of ring finger, first joint of little finger And a bending angle 318 of the second joint of the little finger. Also, 320, 321, 3
Reference numeral 22 represents information on the position, direction, and bending of the finger at times t1, t2, and tn, respectively. In this way, the motion in sign language is represented as time-series data including the hand position 301, the hand direction 305, and the finger bending 309.

FIG. 4 is a storage format diagram of a parameter for recognizing an operating element stored in the operating element parameter memory of the present invention. In FIG. 4, a motion element name 401 represents the name of a motion element whose parameter is used for recognition processing, a parameter number 402 represents the number of parameters used for recognition of the motion element, and 403 to 405 represent Represents a parameter. Also, parameter name 4
06 is a name representing the meaning of the parameter, and parameter 407 represents the value of the parameter actually used in the recognition processing. Specifically, the parameter is a numerical value indicating the speed of hand movement, linearity, a range indicating hand position, and the like. FIG. 5 is a format diagram of the sign language word dictionary stored in the sign language word dictionary memory of the present invention. In FIG. 5, the sign language word name 501 represents the sign language word name represented by the combination of the action elements. The operation type 502 represents whether the entire operation represented by the combination of operation elements is a single operation performed only once or a repeated operation repeated several times. In case of repeated operation, also specify the number of repetitions. The sequential combination number 503 represents the number of operations that are sequentially combined. Sequential operation 50
4 to 506 represent respective operations that are sequentially combined. The number of simultaneous combinations for each sequential operation is 50
7, motion elements 508, 510, 512, and weight values 509, 511, 513 representing the importance of each motion element are stored. The simultaneous combination number 507 is the number of operation elements that are simultaneously combined to represent each sequential operation.

Next, the sign language recognition method of the present invention will be described in detail with reference to FIGS. 6 to 17. FIG. 6 is a flowchart of the processing of the arithmetic unit for recognizing the motion data input in the format of FIG. In the recognition process, first, in step 601, it is determined whether or not the motion data read from the sign language input device 201 is the last, and if it is not the last, the process proceeds to step 602. If the motion data is the last one, the recognition process ends. Step 6
In 02, the operation data for one time is read from the sign language input device 201. In the next step 603, recognition processing of each motion element is performed on the input motion data. The operation of step 603 will be described in detail with reference to FIG. In the recognition of each motion element, since there are many motion elements to be recognized, it is first determined in step 701 whether there is a motion element to be recognized. If there is an operation element for performing the recognition process, the process proceeds to step 702, and if not, the process ends. In step 702, one motion element for which the recognition process is performed is selected from motion elements for which the recognition process is not performed. Next, in step 703, recognition processing of the selected motion element is performed. In the recognition processing, as described above, it is determined whether or not the corresponding motion element can be recognized using the parameters.

FIG. 8 is a diagram showing a list of motion elements in sign language, FIG. 9 is a diagram for explaining a range of motion elements “direction”, and FIG. 10 is a range of motion elements “position”. And FIG. 11 is a diagram for explaining a method of recognizing the motion element “direction”. For example, numbers such as 1 to 5 of operation elements, katakana characters such as aiueo, symbols such as 2B and 3A, and kanji such as medicine and sa are recognized by the shape of the hand. Also, the directions of up, down, left, right, etc. are recognized by the direction of the hand, body parts such as the mouth, head, etc. are recognized by the position of the hand, and curves such as static, straight lines, and circles are the loci of movement. Recognized actions such as pinching and hooking are recognized by the relationship of the fingers, and types such as large, medium, and small are recognized by the magnitude of the motion. There are various types of motion elements to be recognized, such as the shape, direction, position, and movement of a hand, as shown in FIG. The method of recognizing these motion elements is a type of pattern recognition, like voice recognition and image recognition. As a pattern recognition method, three types of methods are widely known: a pattern matching method, a statistical method, and a procedure method. For the recognition of the motion element, the optimum recognition method among these recognition methods can be used according to the property of the motion element. For example, with respect to the shape of the hand, there are many parameters that determine the shape, and there are many cases where subtle relationships between parameters are required. Therefore, a method that collects a large amount of data and performs recognition using a statistical method is suitable. There is. Regarding the direction and position of the hand and the relationship between the two hands,
The procedural method is suitable because the number of parameters is small and the range of parameters can be clearly determined. For example, the directional motion element is shown in FIG. 9, and the position motion element is shown in FIG.
, The range can be determined respectively.

In this case, for example, in order to recognize the motion element related to the direction, regarding the input direction data and its range,
Recognition can be performed using an evaluation function as shown in the following (Equation 1).

[Equation 1] For example, when the direction to be obtained is “up”, the reference vector (x0, y0, z0) in (Equation 1) represents the vector 1101 in FIG. Also, input vector 1102
Then, Ad in (Equation 1) represents an angle 1103 formed by the reference vector and the input vector. Then, if the value of TH2 is in the shaded range 1104,
The evaluation value D1 obtained by (Equation 1) is given a value larger than 0 in the shaded range 1104.
Furthermore, a procedural method can be used in the case of recognition of motion elements related to hand movements, and in the case of motion elements that can be relatively easily ruled like linear and circular motions. FIG. 12 is an operation flowchart showing a method of recognizing the motion element “straight line”, and FIG. 13 is an explanatory diagram of an angle evaluation method in recognition of the motion element “straight line”. For example,
The operation for recognizing the straight line can be performed by the processing of the flow shown in FIG. That is, in order to obtain a straight line, first in step 1201, which direction the straight line is obtained in is determined. Next, in step 1202, the start point of the straight line is obtained, and in step 1203, the end point of the straight line is obtained. In linear hand movements,
At the beginning and end of the movement, the movement tends to stop or become slow. Therefore, the start point and the end point can be obtained by calculating the motion speed of the hand and obtaining the minimum value thereof. In step 1204, the angle between the straight line to be obtained and the line connecting the start point and the end point is obtained. That is, the straight line 1301 in FIG.
An angle 1303 between the line 1302 and the line 1302 is obtained.

In step 1205, it is determined whether the angle is less than or equal to a threshold value. If it is less than or equal to the threshold value, step 1 is performed.
Proceed to 206. If the angle is larger than the threshold value, the process ends. In step 1206, it is checked how much the trajectory from the start point to the end point deviates from the straight line, and the maximum value of the deviation is obtained. FIG. 14 is an explanatory diagram of a deviation evaluation method in recognition of the motion element “straight line”. That is, FIG.
At the locus 1 between the start point 1401 and the end point 1402 at
A perpendicular line 1 from 403 to a straight line 1404 connecting the start point and the end point
405, and the maximum value 1406 of the length is calculated. In step 1207, it is determined whether or not the calculated maximum value of the deviation is less than or equal to the threshold value. On the other hand, if the deviation is larger than the threshold value, the process ends. In step 1208, the evaluation value of the detected straight line is calculated from the angle obtained in step 1204 and the deviation obtained in step 1206. For the calculation of the evaluation value, for example, a calculation formula such as (Equation 2) can be used.

[Equation 2] Here, the angle formed by the straight line obtained by Ang and the straight line connecting the start point and the end point, THa2 is the threshold value of the angle, THd2 is the threshold value of deviation, d21 is the evaluation value for the angle, d22 is the evaluation value for the deviation, and D2 is the detected straight line. It is an evaluation value. In this way, for arcs, wavy lines, circles, spirals, and vibrations, the detection process can be procedurally recognized in the same manner. Further, when it is difficult to form a rule like a free curve, a method based on pattern matching can be used. As a method of pattern matching, a method well known in voice recognition and image recognition can be used. For example, DP collation (Yoshiaki Shirai, "Pattern Understanding" Ohmsha, 1987) can be used. Also, the parameters for recognizing motion elements are
The parameter of the corresponding operating element is read from the memory 202 and used. As a method of recognizing motion elements, in addition to recognizing each motion element by one kind of recognition method, recognition processing is performed using a plurality of recognition methods, and the best value among the evaluation values obtained as a result is calculated as the motion. It may be used as the recognition result of the element.

After performing the recognition process of the selected motion element in step 703 in the motion element recognition process flow shown in FIG. 7, it is determined in step 704 whether the motion element subjected to the recognition process is recognized. This judgment is made using an evaluation formula such as (Equation 1) or (Equation 2).
It can be determined by whether the evaluation value is greater than 0. If the evaluation value is 0, it can be determined that no motion element has been detected. In that case, step 70
Returning to 1, the recognition processing of other motion elements is performed. If the evaluation value is greater than 0, it is determined that the motion element has been detected, and the motion element information including the start time, end time, and evaluation value of the motion element detected in step 705 is shown in FIG.
It is stored in the memory 208 of the recognition result of the motion element.
After the process of step 705 is completed, the process returns to step 701, and the recognition process of other action elements is performed. After recognizing each motion element in step 603 in the processing flow of the arithmetic unit shown in FIG. 6, it is determined in step 604 whether a new motion element is detected. For this judgment,
This can be done by referring to the contents of the memory 208 in FIG. 2 and checking whether or not a new operation element is stored. If a new motion element is stored, the process proceeds to step 605, and the sign language word is recognized. On the other hand, if no new motion element is stored, the process returns to step 601 and the process for the next data is performed.

FIG. 15 is an operation flowchart of a sign language word recognition method according to an embodiment of the present invention, and FIG. 16 is a diagram for explaining overlap between operation elements. Regarding the sign language word recognition method in step 605 of FIG. 6, FIG.
This will be described with reference to FIG. The sign language words are expressed by the simultaneous and sequential combination of motion elements, and their information is stored in the sign language word dictionary of the memory 207. First, in step 1501, a sign language word including the motion element newly detected in step 603 of FIG. 6 is searched from the sign language word dictionary stored in the memory 207.
Next, in step 1502, it is determined whether or not there is a sign language word to be recognized, and if there is a sign language word, the process proceeds to step 1503. If there is no sign language word, the process ends. In step 1503, one sign language word to be recognized is selected from the sign language words searched in step 1501. Next, in step 1504, the memory 208 storing the motion element recognition result is searched for a motion element forming the sign language word. In step 1505, it is determined whether or not all the motion elements necessary for the sign language word to be recognized are found. If found, the process proceeds to step 1506. If all the motion elements are not complete, the process returns to step 1502 to recognize another sign language word. In step 1506, a check is performed on the operation elements that are simultaneously expressed among the operation elements, that is, the operation elements 1 to n described during the sequential operation of each of 507 to 513 in FIG. The simultaneous check of motion elements is performed by obtaining the overlap of time ranges of necessary motion elements.

In this overlap, for example, three motion elements are
When it exists as shown in FIG. 16, it corresponds to the range 1601. When there are n motion elements, this overlap can be obtained by (Equation 3).

(Equation 3) In FIG. 16, since the time axis advances from the smaller one on the left side to the larger one on the right side, the maximum value of the starting points of the motion element i becomes the start point of the overlap and the end point of the motion element i, as in the equation (3). The minimum value of these is the end point of the overlap. Step 15
At 07, it is determined whether or not the detected positions of the motion elements that are simultaneously expressed are within the allowable range. This determination can be performed by checking whether or not the overlapping range of the motion elements obtained by (Equation 3) is greater than or equal to a certain threshold. If the detection range of the simultaneous operation elements is within the allowable range, the process proceeds to step 1508. If it is out of the allowable range, the process returns to step 1502, and another sign language word recognition process is performed. In step 1508, the relationship between the operations expressed sequentially, that is, the sequential operation 504 in FIG.
It is checked whether or not 505 and 506 are detected within the allowable range in that order. For this purpose, the overlapping range of the simultaneously expressed motion elements obtained in step 1506 is used. The overlapping range is set as the detection range of each sequential operation, and the temporal relationship between the ranges is examined. The relationship between the ranges can be evaluated by, for example, a ratio of a range of a certain motion and a range of subsequent motions and an overlap between the motions or a size of a gap. With such a definition, an expression such as (Equation 4) can be used as the evaluation value of the relationship between the i-th and (i + 1) -th sequential operations.

[Equation 4] In (Equation 4), the ratio between the size of the gap or overlap between the i-th and (i + 1) -th sequential operations and the size of the sequential operation range having the smaller detection range is obtained. This is the ratio of 1701 and 1702 in FIG. FIG. 17
FIG. 6 is a diagram illustrating a relationship between sequential operations.

In step 1509, step 1508
It is determined whether or not the relationship between the sequential operations obtained in step 1 is below a threshold value. If the relationship between all the sequential operations is less than or equal to the threshold value, the process proceeds to step 1510. If it is larger than the threshold value, the process returns to step 1502, and another sign language word recognition process is performed. In step 1510, in order to notify the user that the sign language word has been recognized, the word name of the voice word corresponding to the sign language word, that is, the sign language word name 501 in FIG.
Is output. FIG. 18 is a hardware configuration diagram of a sign language recognition apparatus showing a second embodiment of the present invention, and shows a case where two arithmetic units are used. The hardware configuration may be one having two arithmetic units as shown in FIG. 18 in addition to the one arithmetic unit shown in FIG. In this case, the arithmetic unit a180
1 recognizes a motion element, and the arithmetic unit b 1802 recognizes a sign language word, but the arithmetic units are connected via the memory 1803. The recognition algorithm of each arithmetic unit is a process obtained by dividing the recognition process in FIG. FIG. 19 shows the arithmetic unit (a
It is a flowchart of the recognition processing of 1801). FIG.
Then, in step 1901, the manual operation input device 201
It is determined whether or not the motion data to be read from is the last one, and if it is the last, the processing ends. If it is not the last, the process proceeds to step 1902 to read the data for one time from the manual operation input device 201. Next step 1903
Then, the process of recognizing each operation element is performed. The recognition process of each operation element can be performed according to the flowchart of FIG. 7.

FIG. 20 shows the arithmetic unit (b1 in FIG. 18).
It is a flowchart of the recognition process of 802). In FIG. 20, first, in step 2001, the memory 18 of FIG.
The motion element recognition result in 03 is searched, and the newly detected motion element is searched. Next, step 200
In step 2, it is determined whether or not a new motion element is detected as a result of the search for the motion element, and if there is a new motion element, the process proceeds to step 2003. If no new motion element is detected, the process proceeds to step 2004, and it is checked whether or not the motion element has been searched until the last time of the input data. If the search has been performed up to the last time of the input data, the process is terminated. If not, the process returns to step 2001 to search for a new motion element. In step 2003, sign language recognition processing is performed on the newly detected motion element. The sign language word recognition process can be performed according to the flowchart shown in FIG. FIG. 21 is a hardware block diagram of a sign language recognition apparatus showing a third embodiment of the present invention, in which an image input apparatus is added as a sign language input apparatus. As shown in FIG. 21, an image input device 2102 may be used as a sign language input device in addition to the hand movement input device 2101 for recognizing facial expressions, mouth movements, face movements, and the like. In this case, a method well known in image recognition can be used as a method for recognizing motion elements such as facial expressions and face movements detected from an image. Further, by treating the motion elements detected from the image in the same manner as the motion elements shown in FIG. 8, the sign language word dictionary can use the format shown in FIG. 5 as it is. Further, as the method of recognizing the sign language word, the method shown in FIG. 15 can be used as it is.

[0020]

As described above, according to the present invention,
Since the hand movement pattern is first recognized in units of movement elements and then the recognition results of the movement elements are integrated to recognize the words, the sign language words that include movements that change depending on the context or situation in which they are expressed do not change. Since sign language words can be recognized with emphasis on, and each motion element can be recognized using a method suitable for the nature of each motion element, it is flexible, efficient, and accurate. Can perform good sign language recognition.

[Brief description of drawings]

FIG. 1 is a functional conceptual block diagram of a sign language recognition apparatus showing an embodiment of the present invention.

FIG. 2 is a hardware configuration diagram of a sign language recognition device according to the first embodiment of the present invention.

FIG. 3 is a format diagram of operation data input from the sign language input device of the present invention.

FIG. 4 is a format diagram of parameters used for recognition of motion elements in the present invention.

FIG. 5 is a format diagram of a sign language word dictionary according to the present invention.

6 is a flowchart for explaining a sign language recognition method of the arithmetic device in FIG.

FIG. 7 is a flowchart for explaining a method of recognizing a motion element in FIG.

FIG. 8 is a diagram showing a list of operation elements in the sign language of the present invention.

FIG. 9 is a diagram for explaining a range of an operation element “direction” of the present invention.

FIG. 10 is a diagram for explaining the range of the operating element “position” of the present invention.

FIG. 11 is a diagram for explaining a method of recognizing a motion element “direction” according to the present invention.

FIG. 12 is a flowchart for explaining a method of recognizing a motion element “straight line” according to the present invention.

FIG. 13 is a diagram for explaining an angle evaluation method in recognizing the motion element “straight line” shown in the flow of FIG.

FIG. 14 is a diagram for explaining a method of evaluating a deviation in recognition of the motion element “straight line” in FIG. 12;

FIG. 15 is a flowchart illustrating a method for recognizing a sign language word according to the present invention.

FIG. 16 is a diagram for explaining an overlap between operation elements according to the present invention.

FIG. 17 is a diagram for explaining the relationship between the sequential operations in FIG.

FIG. 18 is a hardware configuration diagram of a sign language recognition device using two arithmetic devices, showing a second embodiment of the present invention.

19 is a flow chart for explaining the operation of the arithmetic device for recognizing the operation element in FIG.

20 is a flow chart for explaining the operation of the arithmetic device for recognizing the sign language word in FIG.

FIG. 21 is a hardware configuration diagram of a sign language recognition device to which an image input device is added as a sign language input device, showing a third embodiment of the present invention.

Claims (11)

[Claims] 1. A sign language input means for converting a hand shape or a movement pattern into an electric signal and inputting it to a recognition means, and a hand movement pattern is divided into movement elements which are constituent elements of movement in advance. A sign language expressed by a combination of motion element recognition means that recognizes each motion element in a sign language word from input hand information, motion element recognition result storage means that stores the recognition result of a motion element, and a symbol that represents a motion element. A sign language word dictionary that stores sign language data representing words, a sign language word recognition means for recognizing a sign language word composed of the motion elements by collating the recognized motion elements with the contents of the sign language data, and the recognized sign language words. A sign language recognition device, comprising: an output unit configured to output a voice word corresponding to a voice or a character. 2. The sign language recognition device according to claim 1, wherein the motion element recognition means independently recognizes each motion element, and stores the recognized motion element in the motion element recognition result storage means. Sign language recognition device. 3. The sign language recognition device according to claim 2,
The sign language recognition device, wherein the motion element recognition means performs recognition processing using a recognition method selected according to the property of each motion element. 4. The sign language recognition device according to claim 3,
The motion element recognition means uses a statistical method for hand shape recognition processing, a procedural method for hand direction and position recognition, and a simple hand movement recognition processing, and a complicated method. A sign language recognition device characterized in that a pattern matching method is used for recognizing various hand movements. 5. The sign language recognition device according to claim 3,
The above-mentioned action element recognition means is characterized in that a plurality of recognition methods available for recognizing the constituent element are used, and the recognition result of the method that gives the best result is used as the recognition result of the constituent element. Sign language recognizer. 6. The sign language recognition device according to claim 2,
The sign language recognition device characterized in that the motion element recognition means and the sign language word recognition means operate independently, and the motion element recognition means and the sign language word recognition means are connected via a storage means of a recognition result of the motion element. . 7. The sign language recognition device according to claim 1, wherein
The sign language word dictionary that stores the sign language data is a symbol that represents a motion element necessary for expressing the basic motion of each sign language word, and the degree of change of the motion element depending on the context or situation in which each sign language word is expressed. A sign language recognition device, which stores sign language data including a weight value for each motion element based on. 8. The sign language recognition device according to claim 1, wherein
The sign language word dictionary for storing the sign language data is a sign language recognition device characterized by storing, for each sign language word, a symbol having only a motion element that does not change depending on a context or a situation in which the sign language word is expressed. 9. The sign language recognition apparatus according to claim 1,
The sign language word recognition means searches the recognition result of the motion element necessary for the sign language to be matched when the recognition result of the motion element and the sign language data are matched, and the temporal relationship between the searched motion elements is sign language. A sign language recognition device characterized by performing verification by checking whether or not the temporal relationship between motion elements stored in data is the same. 10. The sign language recognition apparatus according to claim 9, wherein the sign language word recognition means compares the recognition result of the motion element with the sign language data, and detects the motion element when a new motion element is recognized. A sign language recognition device characterized by performing only matching of sign language data including. 11. A sign language input means for converting a shape or movement pattern of a hand into an electric signal and inputting it to a recognition means, an image input means for converting an image into an electric signal and inputting it, and a motion pattern of a hand. A motion element recognition means that recognizes each motion element in a sign language word from the input hand information in advance by dividing the motion element, which is a constituent element of the motion, and a facial expression, a mouth movement, and a face in sign language from the input face image. Facial expression motion recognition means for recognizing the motion element of the motion of the human body, the sign language word dictionary that stores the sign language data representing the sign language expressed by the motion elements of the hand motion, facial expression, mouth movement, and face movement, and the recognized motion element. Sign language recognition characterized by having sign language word recognition means for recognizing sign language by collating the contents of sign language data, and output means for outputting a voice word corresponding to the recognized sign language word by voice or characters. Location.