JPH06230716A - Device and method for finger language translation - Google Patents

Abstract

PURPOSE:To provide a finger language translation device by which a dialogue between an aurally handicapped person and an aurally healthy person can be quickly and accurately performed without performing the interpret processing of finger language, and the aurally healthy person can learn the finger language by the dialogue. CONSTITUTION:A response text prediction part 15 estimates a response text by the aurally handicapped person for the input of the analysis result of the input of the natural language of the aurally healthy person from the result, and provides a result of conversion to the finger language by a finger language conversion part 14 to both the aurally healthy person and the aurally handicapped person with an input text. A simplified input part 16 performs the display of an image by which a routine text can be associated and that of the routine text for a position designated on the image.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a sign language interpreter and method for interpreting between a deaf person who mainly uses sign language as a communication means and a normal hearing person who uses natural language.

[0002]

2. Description of the Related Art Interpretation between a hearing-impaired person and a hearing-impaired person requires interpretation both from the hearing-impaired person to the hearing-impaired person and from the hearing-impaired person to the hearing-impaired person. Conventionally, the former recognizes a sign language motion of a hearing impaired person taken by a camera or the like, thereby interpreting from a hearing impaired person to a normal hearing person, as described in, for example, Japanese Patent Application Laid-Open No. 2-144675. . The latter interprets a natural language input from a voice or a key and analyzes the natural language to interpret from a hearing person to a hearing impaired person.

[0003]

Of the above-mentioned conventional techniques,
First, the recognition method of sign language used when interpreting from a hearing-impaired person to a hearing-impaired person, and the recognition of a speech used when interpreting from a hearing-impaired person to a hearing-impaired person differ depending on the person (for example, depending on the dialect). However, there is a problem that a sufficient recognition result cannot be obtained, and thus a device having sufficient performance cannot be realized.

For this reason, for example, the most reliable input method is for a hearing person to input a natural language sentence with unambiguous keys. However, except for those who perform key operations on a daily basis, There is a problem that input from is very troublesome. In particular, it is very annoying to enter sentences that are frequently used on a daily basis.

Further, ideally, a hearing person understands sign language,
Although it is desirable to be able to communicate with a deaf person by sign language, in the conventional technology, sign language learning and sign language communication must be performed by separate devices, and the sign language interpreter system itself has a function that allows a hearing-impaired person to learn sign language. Do not consider.

An object of the present invention is to improve the interactivity of a sign language interpreting device for conducting a dialogue between a deaf person and a hearing person.

[0007]

[Means for Solving the Problems] In order to achieve the above object, a natural language sentence input from a normal hearing person is analyzed, the analyzed result is converted into a sentence expressed in sign language, and the sentence expressed in sign language is converted. Is displayed to both a hearing-impaired person and a hearing-impaired person, and one or more response sentences to the input sentence are predicted, and the response sentence is also in a plurality of forms such as natural language, sign language, and animation. When one of the response sentences displayed on both sides and one of the two is designated, the selection can be confirmed on the other side.

By displaying an object (for example, a human body) to be interacted with and designating some part (for example, a head) of the displayed object (human body) with a mouse or the like, Alternatively, one or more question sentences (for example, whether there is heat) that can be associated from the designated position or a sentence for setting attributes are displayed, and one of the sentences displayed by a hearing-impaired person with a mouse or the like. By selecting, the sentence is processed as an input sentence from a hearing person.

The analysis result of the natural language sentence input by the normal hearing person is displayed, and if there is no intention from the normal hearing person whether the analysis result is correct or not, it is determined that the analysis result is correct. continue.

[0010]

First, by displaying the response sentence to the input, the hearing impaired person answers one of them, so that unambiguous communication can be realized. Here, for example, if "none" is included in the response sentence, it can be realized without losing flexibility. In addition, since the translation results of the input and the response sentence are also shown to the normal hearing person, even if the hearing impaired person does not select one of the response sentences with the mouse etc. It is possible to understand by comparing with the response sentence. In addition, sign language can be learned through this.

Further, since frequently used input sentences can be taken out and inputted in association with the object to be interacted with, it is possible to perform prompt input without breaking the rhythm of conversation.

Further, when confirming the analysis result of the natural language sentence, if the result is correct, nothing needs to be done, so that the troublesome operation can be reduced.

[0013]

Embodiments of the present invention will be described below with reference to FIGS.

1 and 2 are block diagrams of a sign language interpreter showing an embodiment of the present invention. As shown in FIG. 1, the hearing-impaired person and the hearing-impaired person touch each other with the touch panels 2 and 6, the mice 3 and 7,
Using the keys 4 and 8 and the displays 5 and 9, dialogue is performed via the sign language interpreter processing unit 1. Although not shown in the figure, a sound input device and a sound output device are provided on the side of a normal hearing person so that the sign language interpretation processing unit 1 recognizes the sound of the normal hearing person.
It may be possible to communicate with the hearing impaired person by synthesizing information from the hearing impaired person.
It is also conceivable that the deaf person has a glove or a TV camera for inputting data, and the sign language interpreting processing unit 1 recognizes the sign language operation of the deaf person to communicate with a hearing person.

FIG. 2 shows the configuration of the sign language interpreter processing unit 1, which accepts inputs from the touch panels 2 and 6, the mice 3 and 7, the keys 4 and 8, and the displays 5 and 9.
Is displayed on the interface unit 11.

In FIG. 2, when a sentence in a natural language is input from a normal hearing person through the interface unit 11, the natural language analysis unit 12 analyzes the sentence and sends the analysis result to the analysis result confirmation unit 13. The analysis result confirmation unit 13 confirms the analysis result with a hearing person, and outputs the result when the analysis result is approved. The sign language conversion unit 14 converts the input analysis result into a sign language, and the conversion result is displayed on the displays 5 and 9 through the interface unit 11. The response sentence prediction unit 15 (question-response correspondence dictionary 15
1), the simple input unit 16 and the like will be described later.

The natural language analysis unit 12 can be realized by a conventional natural language processing technique. Next, the analysis result confirmation unit 13
Confirms the analysis result in the procedure as shown in FIG. 3 according to the analysis result. First, in step 1301, it is determined whether the sentence could be uniquely interpreted. If there is only one interpretation, an inquiry is made as to whether the interpretation is correct in step 1302, and if a response that the interpretation is correct is given in step 1303, the interpretation is output and the processing ends.
If there is a response to the mistake, step 13
At 04, re-entry or correction of entry is requested. here,
In the response to the confirmation of the interpretation, if there is no right or wrong response within a certain period of time, it is determined to be correct.

Next, when there are a plurality of interpretations, the reliability of each interpretation is obtained in step 1305. For example, the interpretation of "Do you have a good appetite?" Can be interpreted as "Can you eat?" Or "Are you hungry?" Here, assuming the case of a medical examination in a hospital, the latter interpretation is unnatural, and the reliability is lower than the former interpretation. In step 1306, it is checked whether or not there is a highly reliable interpretation, and if there is, an inquiry is made in steps 1307 and 1308 as to whether or not the highly reliable interpretation is correct. Here, if there is a response to the error, the rest of the interpretation with low reliability that did not make an inquiry is confirmed. Step 130
At 9, all interpretations are presented and the intended interpretation is selected. If there is no intended interpretation (step 1310), re-input is requested in step 1311.

In sign language, for example, "like" and "~"
"I want you to do" is also expressed by a sign language operation expressing "like" (hereinafter, this is called a sign language word, and a word normally defined in natural language is called a voice word). That is,
A plurality of voice words are often expressed by one sign language word. On the contrary, it may be necessary to use sign language words that have the same character face but different sign language words. For example, in FIG.
2 shows an example of the sign language conversion dictionary 141 of FIG. 2, but when the voice word “Aru” is “I have an appetite”, the sign language word “OK” has “Fever etc.” You need to use the sign language word "How are you?" Sign language converter 1
4 is a sign language conversion dictionary 1 when "I have an appetite" is input.
Based on 41, it is converted into a sign language word string of "eat (noun), okay". Then, the image data (or animation) of the gesture that actually expresses each sign language word is retrieved from the sign language data 142 and displayed on the displays 5 and 9 through the interface unit 11.

Here, in the sign language data 142, for example,
Image data of a gesture of a sign language word taken by a glove for inputting data (for example, a time change pattern of position data of each joint of a finger) is stored together with its attribute data as shown in FIG. As the attribute data, headline and part-of-speech of a sign language word for searching data, start and end point coordinates (X, Y, Z coordinates) when data is collected, and displayed center position coordinates, display summary position described later. One of
The center position coordinates displayed when moving to one of the outline positions and a flag indicating the directionality of the data are included.

Here, the center position coordinates are, for example, the sign language word "mouth" is drawn around the mouth with a finger, and the center position coordinates of this word are the coordinates in the middle of the mouth. . Actually, when the sign language is generated, the position of the wrist or the finger is a real position that can be detected, but the center position coordinate is a virtual position that does not exist. However, this central position plays an important role in grasping the general position of sign language when we express it. Therefore, the center position coordinates are registered as a reference. Similarly, the start / end coordinates are not the specific positions of the wrist or finger, but the center position of the expression at the start or end of the word. Next, the direction is
The degree of change in the position or direction at the time of display is classified as follows, for example.

(1) A word that is displayed in the registered position (think, mouth, etc.) (2) A word that moves only in the display position (he, house) (3) A word in which the display position and the direction of the palm change ( (Pain) (4) A word that moves with the display position and direction, but the movement of the starting point is small (passing) (5) A word that moves with the display position and direction and also moves the starting point greatly (speaking) The data of the hand of the other hand is stored, but in the case of sign language in which only one hand is expressed, nu
Stores data such as ll.

The sign language conversion unit 14 generates sign language image data in the following procedure. Generally, in sign language, the expression position of the character or thing (hereinafter referred to as an event) of the sentence to be expressed is first set, and the direction of the action of the action exerted between the events is expressed by a directional verb. (It's called a relational expression here). The placement strategy unit 1401 of FIG. 6 determines the expression position of each word by using the placement rule 1404 with the information of the sign language word string converted into the sign language word by the word conversion unit 1400 and the information of the dependency relation between the words as input. To do. The display position determined here is a general outline position, and for example, the horizontal direction (right, middle, left),
It is a position where the vertical direction is divided into (top, middle, and bottom) and the front-rear direction is divided into (front, back). Therefore, specify in the form of (horizontal direction, vertical direction, front-back direction). For example, (middle, bottom, front), which is the above-mentioned display outline position. Then, the center point is set in advance for each display outline position. As shown in the structure of the sign language data 142, the direction is described in each word. First, for a word that does not move in a direction like a noun (classification 2), its display outline position is simply determined. Next, a verb such as "pass" changes its motion direction (classification 4), and an adjective or verb such as "pain" or "pain" changes its expression position (classification 3). In the former case, a triplet structure consisting of a word name and an event at the starting point and an event at the end point of the action is generated, and in the latter case, a structure of a word name and its nominative pair is generated. For example, in the sentence "He gave her a book," (Give, book, she), and in the sentence "Head hurts," it hurts (head). Here, in “speaking” (classification 5) that also involves movement of the starting point, for example, the sentence “he speaks to her” is converted to (speak, he, she).

Next, the real coordinate determination unit 1402 searches the above-mentioned sign language data 142 and determines the coordinate position to be actually displayed. First, in the case of a word expressing an event, the word data is retrieved from the sign language data 142, and the center position coordinates thereof are aligned with the center point of the display outline position designated by the placement strategy unit 1401 to realize a spatial expression. To do. That is, the movement amount of the coordinates is obtained from the center point of the specified display outline position and the center position coordinates of the word. Coordinates such as the start and end points can be calculated based on this movement amount.

Next, among the words expressing the relationship, the words whose movement direction changes are passed in the form of (word name, event at start point, event at end point). Therefore, the word data is searched and calculated so that the start point and end point coordinates of the word data are located on the line connecting the center positions of the start point event and the end point event, and then converted. Also, the words that change only in position are
Passed in the format (word name, nominative). In this case, it is expressed in the central position coordinates where the nominative event is expressed. Further, the word “think” (classification 1) and the like are words that do not change, and thus are expressed as they are retrieved from the sign language data 142. In addition, in the case of one-handed sign language, it is also determined which hand is used. Furthermore, "he (left hand)",
When expressing "she (right hand)" in the expressions "she (right hand)", "book (both hands)", and "raising (both hands)", the left hand will keep displaying "he" or in the standby position. Also decide whether to return.

The interpolation processing unit 1403 is used by the real coordinate determining unit 14
Based on the sign language word position determined in 02, interpolation processing is performed so that image data of adjacent sign language words can be smoothly connected. Here, not only the hands but also the models of the arms and the body are constructed by using the conventional computer graphic technology. Then, the sign language word is based on the movement data of the hand or the like of the sign language word taken by the above-mentioned data globe,
It is displayed by operating the above model. As a result, the interpolation process between the image data of the sign language words is also performed based on the model. That is, it can be realized by moving the hand or arm of the model from the position of the hand or arm at the end of the previous sign language word to the position of the hand or arm at the start of the subsequent sign language word. Here, in the interpolation between the image data of the sign language words, if it is not possible to connect smoothly to the position determined in the outline position setting in the previous step, the determined position can be corrected and then connected. In this way, the sign language image display can be realized.

The response sentence prediction unit 15 of FIG. 2 retrieves the response sentence from the question-response correspondence dictionary 151 based on the analysis result of the natural language analysis unit 12. FIG. 7 shows an example of the question-answer correspondence dictionary 151. In the figure, what is shown in parentheses is the notation of sign language words. Questions and their responses are stored here, and the analyzed question is used as a key.
It is possible to extract the response sentence of the corresponding voice word and sign language word. This result is converted into image data by the sign language conversion unit 14, and then, together with the input question as shown in FIG.
It will be displayed to both deaf and hearing-impaired people by means of spoken language and sign language.

In this way, since the translation result of the input and the response sentence are also shown to the normal hearing person, even if the hearing impaired person does not select one of the response sentences with the mouse 7 or the like, even if he / she responds by sign language, A hearing person can understand the response by comparing it with the response sentence displayed in sign language. In addition, through this, sign language can be learned while using the device.

A hearing person inputs a natural language sentence with a key, but the input from the key is very troublesome. Especially, it is very troublesome to input frequently used sentences one by one. For example, in the case of a medical examination at a hospital, the simple input unit 16 extracts image data of a human body from the target image 161, and displays it on the display 5. When the hearing-impaired person specifies any position on the display 5 with the mouse 3 or the like, the fixed phrase corresponding to the position (specifically, a section of a certain range) is searched from the fixed phrase dictionary 162, and the fixed phrase is displayed in FIG. As in the menu format. When one of these sentences is designated by a hearing person with the mouse 3 or the like, the simple input unit 16 outputs the sentence as an input sentence. Specifically, as shown in FIG. 10, the fixed phrase dictionary 162 registers fixed phrases in pairs with section names. In addition, each section of the section name has actual coordinates on the display 5 (for example, section 1 has a point {X coordinate = 500, Y coordinate = 700} as an upper left corner and width = 10.
A table showing the correspondence with 0, height = 100 section) is registered. Section 1 is set at the position of the head of the human body. As a result, the position of the head on the display 5, for example, {X coordinate =
When a point of 550, Y coordinate = 650} is designated, this point is required to be a point within the section 1 from the above table. Then, by searching the fixed sentence dictionary 162 of FIG. 10 for the fixed sentence paired in the section 1, the display as shown in FIG. 9 is possible.

As described above, since the inquiry in the natural language can be input only by designating with the mouse 3 instead of the key 4, the prompt and correct input is possible.

[0031]

According to the present invention, a dialogue between a deaf person and a normal hearing person can be realized without performing sign language understanding processing, and the normal hearing person can quickly and accurately input a sentence. There is also an effect that you can learn sign language.

[Brief description of drawings]

FIG. 1 is a diagram showing a configuration of a sign language interpreter according to an embodiment of the present invention.

FIG. 2 is a diagram showing a configuration of a sign language interpreting processing unit.

FIG. 3 is a diagram showing confirmation processing of a result of analysis of natural language input.

FIG. 4 is a diagram showing a configuration of a sign language conversion dictionary.

FIG. 5 is a diagram showing a structure of sign language data.

FIG. 6 is a diagram showing a configuration of a sign language conversion unit.

FIG. 7 is a diagram showing a configuration of a question-answer correspondence dictionary.

FIG. 8 is a diagram showing a display example of a translation result of an input by a normal hearing person.

FIG. 9 is a diagram showing a display example of a simple input screen.

FIG. 10 is a diagram showing a configuration of a fixed phrase dictionary.

[Explanation of symbols]

1 Sign language interpreter, 2, 6 ... Touch panel, 3, 7 ...
Mouse 4,8 ... Key, 5,9 ... Display, 11 ...
Interface unit, 12 natural language analysis unit, 13 analysis result confirmation unit, 14 sign language conversion unit, 141 sign language conversion dictionary, 142 sign language data, 15 response sentence prediction unit, 151
Question-response dictionary, 16 simple input unit, 161, target image, 162 fixed phrase dictionary.

Claims (6)

[Claims] 1. A means for analyzing a natural language sentence input by a hearing-impaired person, a means for converting the analyzed result into a sentence expressed in sign language, and a sentence expressed in sign language and the input natural language sentence. Means for both the hearing-impaired person and the deaf person, means for predicting a response sentence to the sentence from the analysis result of the natural language sentence, and the response sentence in a plurality of forms such as a natural language sentence and a sign language image. A sign language interpreting device comprising means for displaying to both a hearing person and a hearing impaired person, and means for confirming the displayed response sentence. 2. A sign language interpreting apparatus according to claim 1, wherein a means for predicting a plurality of response sentences to the sentence from an analysis result of the input natural language sentence, and a plurality of the response sentences for the natural language sentence and the sign language. A sign language interpreter having means for displaying to both a hearing person and a deaf person in a plurality of forms such as images, and means for designating one of the plurality of displayed response sentences. 3. The sign language interpreting device according to claim 1, further comprising means for displaying an object to be interacted with, and by designating a part of the displayed object with a mouse or the like, a question sentence about the part is displayed. A sign language interpreting device, which displays a sentence for setting attributes and attributes, and inputs the displayed sentence as a natural language sentence from a hearing-impaired person. 4. The sign language interpreting apparatus according to claim 1, further comprising means for displaying an analysis result of a natural language sentence input by a normal hearing person, and whether the analysis result is correct within a certain time from the normal hearing person. A sign language interpreter characterized by continuing the processing assuming that the analysis result is correct when there is no intention display. 5. A sign language interpreter according to claim 1, wherein the sign language image data is stored in a form in which it can be discriminated into word data that needs directionality and word data that does not need directionality, and display of the image data of words. A sign language interpreter having a sign language image generation means for determining the detailed position after determining the outline position of the. 6. A natural language sentence input from a normal hearing person is analyzed, the analyzed result is converted into a sentence expressed in sign language, and the sentence expressed in sign language and the input natural language sentence are regarded as a normal hearing person. It is displayed to both deaf and hearing-impaired persons, and the response sentence to the sentence is predicted from the analysis result of the natural-language sentence. A sign language interpreting method characterized by displaying and confirming the displayed response sentence.