KR20060133190A - Sign language phone system using sign recconition and sign generation - Google Patents

Abstract

A two-way language phone system for the recognition and generation of sign language is provided so that aurally handicapped persons can lead leisure life through a video phone system program or obtain information required for social activities or education. A two-way language phone system for the recognition and creation of sign language consists of a system processing part(100), a sign language input part(110), a sign language output part(120), a communication line part(130), and a DB group(140). The system processing part(100) comprises a sentence generation part(200), a voice generation part(400), a sentence creation part(500), and a sign language generation part(300). The sign language input part(110) recognizes a sign language operation or a sign language form through a camera and provides corresponding sign language image data to the sentence generation part(200). The sentence generation part(200) searches the sign language DB of the DB group(140) for a sentence corresponding to the sign language image data provided from the sign language input part(110). The voice generation part(400) obtains the phonemes corresponding to the sentence, provided from the sentence generation part(200), from the voice DB of the DB group(140), combines them into a voice, and provides it to the communication line part(130). The sentence creation part(500) searches the voice recognition DB of the DB group(140) for a sentence corresponding to the voice provided from the communication line part(130) and provides it to the sign language generation part(300).

Description

Sign Language Phone System using Sign Recconition and Sign Generation}

1 is a block diagram showing an embodiment of a two-way sign language telephone system implementing sign language recognition and sign language generation according to the present invention;

FIG. 2 is a block diagram illustrating an embodiment of a sign language generating unit shown in FIG. 1;

3 is a block diagram illustrating an embodiment of a sentence generator shown in FIG. 1;

4 is a block diagram showing an embodiment of a voice generator shown in FIG. 1;

5 is a block diagram illustrating an embodiment of a sentence generation unit illustrated in FIG. 1;

6 is a flow chart showing step by step the operation of the consonant transducer shown in FIG.

7 is a view showing the dependency of sign language morpheme / 3D sign language animation DB, sign language word DB, and hand-shaped DB,

FIG. 8 is a diagram illustrating a collection of syllables and lips before consonant conversion in the consonant converter shown in FIG. 2;

9 is a view showing a variety of lips according to the operation of the consonant transducer shown in FIG.

10 is a view showing a head motion DB,

FIG. 11 is a diagram illustrating various face expressions of a 3D speaker model according to face expression information of a sign language word DB.

12 is a diagram showing hand movement data stored in the hand movement DB;

13 is a view showing hand directions corresponding to hand posture data stored in a hand posture DB;

14 is a view showing a hand shape corresponding to the hand posture data stored in the hand posture DB;

Fig. 15 is a view showing the hand position image subdivision and hand position data contents of the called party;

16 is a diagram illustrating hand position information values generated by dividing a face of a talker's face and segmenting the face of a talker when the hand is located in the face of the talker;

17 is a diagram showing the details of a pixel configuration of a pixel DB;

18 is a view showing the constitution of consonant pixels in the hydration DB;

19 is a view showing the configuration of a collection pixel of a hydration pixel DB;

20 is a view showing the contents of a character array of a pixel DB;

21 is a view showing the deaf mode screen layout,

Fig. 22 is a diagram showing a dry subject mode screen layout.

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a Sign Language Phone System that implements sign language recognition and sign language generation. In particular, when a deaf person makes a call to a hearing person, a sign language feature expressed by a deaf person may be extracted. Refer to) to express the sign language by sentence or voice, and when the callee makes a call to the hearing impaired person, the caller extracts the spoken voice and converts it into a letter and converts it into a sign language expression through various preprocessing algorithms. The present invention relates to a sign language telephone system that promotes access to information for the hearing impaired by expressing a 3D sign language animation with reference to a sign language animation DB.

Through the publications of Korean Patent Laid-Open Publication Nos. 2004-0010945, 2004-0076907, Published Patent Publication No. 0397692 and Published Utility Model Publication No. 0234151, video phones, handwriting recognition phones, hand gesture recognition video communications for the hearing impaired are well known. . However, they do not consider the conversion of sign language-sign language animation and sign recognition-speech synthesis. Also, these documents do not present 3D sign language animation technique, sign language recognition technique, hand position technique, head movement technique, and lip shape transformation technique. Therefore, the specific plan for developing a sign telephone system is not considered.

The present invention has been made to solve the above problems of the prior art, the image data corresponding to the sign language represented by the hearing impaired by the feature extraction, hand motion changes, hand position start point classified and converted into text through the sign language DB search and this The voice can be synthesized again and transmitted to the other party through the communication line, the voice extracted by the person is converted into a sentence by the voice recognizer, processed by a sign language preprocessing algorithm, and combined with the shape of the lips, facial expressions, and head movements. The purpose is to provide a two-way sign language telephone system that can generate a three-dimensional sign language animation.

The present invention for achieving the above object, by using a sentence generation unit for generating a Hangul sentence corresponding to the provided sign language image data, and using a phonetic language corresponding to the Hangul sentence provided from the sentence generation unit Represents a voice generation unit provided to the communication line unit, a sentence generation unit for generating a Hangul sentence corresponding to the voice provided from the communication line unit, and an avatar performing sign language corresponding to the Hangul sentence provided from the sentence generation unit. It characterized in that it comprises a hydration generating unit.

When described in detail with reference to the accompanying drawings an embodiment of the present invention as follows.

Figure 1 is a block diagram showing an embodiment of a two-way sign language telephone system that implements sign language recognition and sign generation according to the present invention, a system processing unit 100, sign language input unit 110, sign language output unit 120, communication line unit 130, and the DB group 140. The system processor 100 includes a sentence generator 200, a voice generator 400, a sentence generator 500, and a sign language generator 300.

In the figure, the sign language input unit 110 recognizes a sign language operation or a sign language by using a camera and provides the sign language image data corresponding to the sign language image unit 200 in the system processing unit 100.

The sentence generation unit 200 searches for a sentence corresponding to the sign language image data provided from the sign language input unit 110 in a sign language DB in the DB group 140 and provides the sentence to the voice generation unit 400.

The voice generator 400 searches for a voice station corresponding to the sentence provided by the sentence generator 200 in the voice DB in the DB group 140, synthesizes the corresponding voice, and communicates with the communication line (telephone network or data communication network) ( 130).

The sentence generation unit 500 searches for a sentence corresponding to the speech provided from the communication line unit 130 in the speech recognition DB in the DB group 140 and provides the sentence generation unit 300.

The sign language generation unit 300 searches for a sign language corresponding to a sentence provided from the sentence generation unit 500 in the sign language DB in the DB group 140, and converts the sign language into data for screen display and provides the sign language output unit 120 to the display language.

The sign language output unit 120 uses the screen display data provided from the sign language generator 300 to display an avatar performing the sign language on the screen output device for the user to view.

2 is a block diagram illustrating an embodiment of the sign language generator 300 shown in FIG. 1. At this time, each DB (211, 212, 213, 214, 215, 221, 222, 223) is configured in the DB group 140 shown in FIG.

In the figure, the morpheme analysis unit 201 in the sign language generation unit 300 analyzes a sentence provided from the sentence generation unit 500 and decomposes it into each morpheme. At this time, the sentence used in the present invention is a Hangul sentence, the morpheme analysis unit 201 analyzes the Hangul sentence and decomposes each Hangul morpheme.

The sign language DB search unit 203 has a sign language word corresponding to each of the decomposed Hangul morphemes provided from the morpheme analysis unit 201 in a sign language morpheme / 3D sign language animation DB 221 and a sign language word DB 222 shown in FIG. 11. When retrieved from, the sign language word DB 222 and the hand-shaped DB 223 extracts modeling information values corresponding to the retrieved word and expresses it in 3D sign language. FIG. 11 is a diagram illustrating various face expressions of a 3D speaker model according to face expression value information of a sign language word DB 222.

The face expression adding unit 206 is connected to the sign language DB search unit 203, and when the sign language DB search unit 203 searches for the sign language word, the facial expression value of the retrieved sign language word is expressed from the expression DB 212. Then, the head movement of the retrieved sign language word is searched from the head movement DB 213 as shown in FIG. 10, and the color motion of sign language is expressed through the head movement. For example, three-dimensional sign language animation can be used to express head movements, such as bowing your head or pushing your head slightly backwards to express a surprised appearance.

On the other hand, the sign language DB search unit 203 has a sign language word corresponding to each decomposed Hangul morpheme provided from the morpheme analysis unit 201 in a sign language morpheme / 3D sign language animation DB 221 and a sign language word DB 222. If it is not found, the unregistered situation is reported to the unregistered processing unit 205, and the unregistered processing unit 205 searches the 3D paper animation DB 211, and the 3D corresponding to the elementary / medium / finality of the sign language word. Express animations.

The consonant converter 202 is connected to the morpheme analysis unit 201 to change the consonant of the initial consonant in the Hangul morpheme to "o" and removes the finality from the morpheme. At this time, the vowels to implement the syllables and lips before the consonant conversion in the consonant transducer 202 is as shown in FIG. Various lip shapes according to the operation of the consonant transducer 202 are as shown in FIG. 9, and the sign language avatar model expresses each vowel resulting from the result of the consonant transducer 202 into each lip shape. In other words, an avatar model that expresses an understanding of sign language information by assisting sign language using a mouth shape is represented.

That is, the lip expression adder 204 retrieves the corresponding lip shape corresponding to the output of the consonant transducer 202 from the lip shape DB 214 to generate the lip shape of the avatar.

The 3D sign language graphic generation unit 207 is connected to the face expression adding unit 206, the unregistered processing unit 205, the lip expression adding unit 204, and the human body model DB 215 to convert the sign language avatar into a 3D sign language animation. Expresses the pronunciation of sign language avatars.

The expression unit 208 converts the 3D sign language animation output of the 3D sign language graphic generator 207 into data for screen display and provides it to the sign language output unit 120.

3 is a block diagram showing an embodiment of the sentence generator 200 shown in FIG. 1. At this time, each DB (311, 312, 313, 314) is configured in the DB group 140 shown in FIG.

In the drawing, the data acquisition unit 301 in the sentence generation unit 200 receives the sign language image data from the sign language input unit 110 to recognize the hand gesture of the receiver in real time and generates color image data to generate the image preprocessor 302. To provide.

The image preprocessor 302 processes the hand image data provided from the data acquirer 301 and classifies the image into hand trajectory data, hand posture data, and hand position data.

The hand motion classification unit 303 generates a hand motion information value by comparing the hand locus data provided from the image preprocessor 302 with the hand motion data retrieved from the hand motion DB 311.

The hand posture classification unit 304 compares the hand posture data provided from the image preprocessor 302 with the data of the hand posture DB 312 to generate a hand posture information value.

The hand position classifier 305 divides the image of the hand position data provided from the image preprocessor 302 into 7X5 lower images based on the center of the receiver and the face of the receiver, and divides the data into the hand position DB ( And hand position information value is generated. In this case, when the finger of the hand position data is in contact with the face portion of the position data, the face image is divided again based on the user's nose, and the divided position image data is compared with the position image data of the hand position DB 313. Generate hand position information.

The sign language data configuration unit 306 is a hand motion information value provided from the hand motion classification unit 303, a hand position information value provided from the hand position classification unit 304, and a hand position provided from the hand position classification unit 305. Comparing the information value to the hand movement information value, the hand posture information value, and the hand position information value stored in the pixel source 314, a text word name corresponding to the name of the word of the pixel in the pixel source 314 is generated. .

The sentence constructing unit 307 recomposes and outputs the sentence using the text word name provided from the sign language data constructing unit 306.

At this time, the hand movement classification unit 303 does not generate a hand movement information value when the hand is in a stationary state without movement. Accordingly, the sign language data configuration unit 306 processes the hand movement data in a stationary state, and when the hand movement data is in the stationary state and the hand posture data contains a unique value, the hydration pixel data is the hydration DB 314. If it matches the consonants / vowels corresponding to the name of the pelvic name, the consonants / vowels are generated, and the generated consonants / vowels are combined to generate proper nouns.

4 is a block diagram showing an embodiment of the voice generator 400 shown in FIG. 1.

In the same figure, the speech synthesis unit 401 searches for and outputs speech data stored in the speech synthesis DB 411 in response to a sentence provided from the sentence generation unit 200 to deliver sign language information of a hearing impaired person to a non-speaker. To be able.

FIG. 5 is a block diagram illustrating an embodiment of the sentence generation unit 500 shown in FIG. 1.

In the figure, the speech recognition unit 501 generates a sentence by appropriately comparing the speech data previously learned in the speech recognition DB 511 in response to the speech provided from the communication circuit 130, and generating a sentence. Provided by 300.

6 is a flowchart illustrating the operation of the consonant transducer illustrated in FIG. 2 step by step.

First, the current word is separated by one syllable in order to implement a sign language movement and a change in the shape of lips (step S2).

Each of the separated syllables is converted into seconds, mid, and finality (step S4).

The consonant is converted to 'o' (step S6).

The vowel is removed from one syllable, and only the vowels are expressed to implement the lip shape (step S8). In this case, when the vowel is a double vowel, the double vowel is separated into a simple vowel.

If the syllable remains, the process is performed again from the step S4 (step S10).

FIG. 7 is a diagram illustrating a dependency relationship between a sign language morpheme / 3D sign language animation DB 221, a sign language word DB 222, and a hand-shaped DB 223. The sign language morpheme according to the operation of the sign language DB search unit 203 is shown. Morphological words input by the 3D sign language animation 221 search are separated into sign language words, and modeling value information obtained from the sign language word DB 222 and the hand-shaped DB 223 is extracted to implement an avatar. .

12 is a diagram illustrating hand movement data stored in the hand movement DB 311.

FIG. 13 is a view showing hand directions corresponding to hand posture data stored in the hand posture DB 312.

FIG. 14 is a diagram showing a hand shape corresponding to the hand posture data stored in the hand posture DB 312.

FIG. 15 is a diagram illustrating the subdividing of the hand position image and the contents of the hand position data of the talker, and shows the hand position information generated by the talker division image and the talker area division when the hand is located outside the face.

FIG. 16 is a diagram illustrating hand position information values generated by dividing a face of a talker's face and segmenting the face of a talker when the hand is located in the face of the talker.

FIG. 17 is a view showing the details of the pixel configuration of the pixel DB 314. The sign language data configuration section 306 provides the hand motion information value provided from the hand motion classification section 303 and the hand detail classification section 304. As shown in FIG. If the hand position information value and the hand position information value provided from the hand position classification unit 306 are received and the three information values match the data previously stored in the pixel DB 314, a corresponding text word is generated.

FIG. 18 is a view showing the consonant pixel configuration of the pixel DB 314. The sign language data configuration unit 306 may generate consonants when all hand movement data is in a stopped state.

19 is a view showing the configuration of the collection pixel of the pixel DB 314. The sign language data configuration unit 306 generates a collection when all the hand movement data is in a stopped state.

FIG. 20 is a view illustrating the arrangement of characters in the sign language DB 314. The collection pixel and the consonant point pixel may be combined and generated in the order of the sign representation of the sign language user. If you configure a lot, you can generate proper nouns by collecting two or more characters.

FIG. 21 is a diagram illustrating a layout of a deaf mode screen of a sign language telephone system, and FIG. 22 is a diagram illustrating a layout of a deaf mode of a sign language telephone system. The sign language animation may be displayed on the counterpart non-talker video screen area.

The technical spirit of the present invention has been described above with reference to the accompanying drawings, but this is by way of example only and not intended to limit the present invention. In addition, it is obvious that any person skilled in the art can make various modifications and imitations without departing from the scope of the technical idea of the present invention.

The present invention is useful for helping the hearing-impaired people to understand the communication through two-way sign phone system that implements sign language recognition and sign language generation, and to provide information necessary for leisure activities or social activities and education through the video phone system program. Can be used.

Claims (5)

A sentence generation unit for generating a Hangul sentence corresponding to the provided sign language image data; A voice generator for synthesizing the voice using a phoneme corresponding to the Hangul sentence provided by the sentence generator and providing the voice to the communication line; A sentence generation unit generating a Hangul sentence corresponding to the voice provided by the communication line unit; And a sign language generation unit for representing an avatar performing a sign language corresponding to the Hangul sentence provided by the sentence generation unit. The method of claim 1, The sign language generation unit, and a morpheme analysis unit for decomposing the sentence provided from the sentence generation unit into each morpheme, When a sign language word corresponding to each morpheme provided from the morpheme analysis unit is provided, a modeling information value corresponding to the provided sign language word is extracted and represented in a 3D sign language graphic, and the respective morpheme analysis units are provided. A sign language retrieval unit for allowing a non-registered processing unit to express a 3D animation corresponding to the elementary / medium / finality of the sign language when the sign language word corresponding to the morpheme is not provided; A face expression adder which is connected to the sign language search unit and expresses a facial expression and a head movement of the sign language word when the sign language search unit includes the sign language word; A consonant converter connected to the morpheme analysis unit to change the consonant of the initial consonant in the morpheme to " o " A lip expression adder for generating a lip shape corresponding to the output of the consonant transducer, And a 3D sign language graphic generation unit connected to the face expression adding unit, an unregistered processing unit, and a lip expression adding unit to express a sign language avatar as a 3D sign language animation. The method of claim 1, The sentence generation unit may include a data acquisition unit receiving the provided sign language image data and generating image data corresponding to a hand gesture of a receiver; An image preprocessing unit classifying hand trajectory data, hand posture data, and hand position data from the hand image data provided from the data acquisition unit; A hand movement classifier configured to compare the hand trajectory data provided from the image preprocessor with the hand movement data and generate a hand movement information value; A hand posture classification unit for generating a hand posture information value by comparing the hand posture data provided from the image preprocessing unit with the provided hand posture data; The image of the hand position data provided from the image preprocessing unit is divided based on the center of the receiver and the face of the receiver, and the hand position data is compared with the hand position image data provided to generate hand position information values. Hand position classification unit, Hand movement information value provided by the hand movement classification unit, hand position information value provided by the hand position classification unit, hand position information value provided by the hand position classification unit, and hand position information value A sign language data constructing unit for generating a text word name corresponding to a sign language word name provided in comparison with a value of a hand and a hand position information; 2. A sign language telephone system embodying a sign language recognition and a sign language generation, comprising a sentence construction unit constituting a sentence using a text word name provided from the sign language data construction unit. The method of claim 3, wherein The hand position classifying unit, when the finger of the hand position data is in contact with the face portion of the position data, the face image is re-divided around the user's nose, and compared with the position image data provided with the divided position image data A bidirectional sign language telephone system implementing sign language recognition and sign language generation, characterized by generating hand position information. The method of claim 3, wherein The hand motion classification unit does not generate a hand motion information value when the hand is stationary and does not move, When the hand movement classification unit does not generate a hand movement information value, the sign language data configuration unit processes the hand movement data in a stationary state, and when the hand movement data is in a stationary state and contains unique values in the hand posture data. When the consonant / vowels correspond to the corresponding consonant names, the consonants / vowels are generated and the proper consonants are generated by combining the generated consonants / vowels. Telephone system.

Images