JP2004254218A - Cellular phone, communication method, and communication program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a cellular phone, communication method and communication program for easily communicating with each other regardless of whether an opposite party is not handicapped in hearing or is handicapped in hearing. <P>SOLUTION: A user uses sign language in front of a camera 1. On the basis of the image recognition result, a sign-language recognizing part 3 recognizes the sign language of the user (person handicapped in hearing). On the basis of the sign-language recognition result, a sign-language/Japanese translation part 5 translates user's sign language into sentences in Japanese and a Japanese/voice translation part 7 translates the sign language into voices. The sentences in Japanese and the voices are transmitted to the terminal of the opposite party, displayed on the terminal of the opposite party and outputted in voice. Besides, when voices from the terminal of the opposite party are received, the Japanese/voice translation part 7 translates the received voices into sentences in Japanese and further translates them into sign-language animation. The sentences in Japanese and the sign-language animation corresponding to the voices of the opposite party are displayed on a display part 9. <P>COPYRIGHT: (C)2004,JPO&NCIPI

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a mobile phone, a communication method, and a communication program for performing communication between hearing-impaired persons or between a hearing-impaired person and a normal hearing person, regardless of the distance to the partner.
[0002]
[Prior art]
Conventionally, when a hearing-impaired person and a hearing-impaired person are in the same place, if the hearing-impaired person can sign, it is possible to converse by sign language. If the hearing person cannot understand the sign language, it is possible to have a conversation by using a handwriting.
[0003]
In addition, when the other party is far away, the spread of mobile phones has recently increased, and it has become possible for a hearing-impaired person to communicate with anyone anytime, anywhere, by telephone. Also, many of these mobile phones have a mail function, and even if the hearing-impaired person uses this mail function, it is possible to communicate with the hearing-impaired or hearing-impaired person. Note that Patent Literature 1 and Patent Literature 2 are disclosed as conventional techniques relating to mobile phones.
[0004]
[Patent Document 1]
JP 2000-224319 A [Patent Document 2]
JP 2001-197457 A
[Problems to be solved by the invention]
By the way, in general, most hearing people cannot understand sign language, and even a hearing-impaired person, only about 10% of 350,000 people in Japan can use sign language. Therefore, when a hearing-impaired person talks with a person who cannot sign, the conversation is mainly written, but writing is time-consuming and troublesome, and it cannot always be said that sufficient communication can be achieved.
[0006]
When the other party is far away, the mail function of the mobile phone may be used as described above. However, the communication function cannot be said to be a sufficient communication function in terms of real time and urgency. If the hearing person wants to talk directly or needs urgency, he or she often communicates by voice instead of email. However, in this case, the hearing impaired person has a problem that it is not possible to perform highly simultaneous communication such as voice communication.
[0007]
As described above, in the conventional mobile phone, the hearing impaired person cannot communicate sufficiently with a normal hearing person or a person who cannot sign, regardless of the distance to the other party.
[0008]
The present invention has been made in view of the above-described circumstances, and regardless of the distance to the other party, the other party is not limited to a hearing-impaired person or a hearing-impaired person, and can easily communicate with a mobile phone, a communication method, and a communication method. It is intended to provide a communication program.
[0009]
[Means for Solving the Problems]
In order to solve the above-mentioned problems, according to the present invention, in a mobile phone that bidirectionally transmits at least voice and text to and from a partner terminal by wireless communication, a photographing means for photographing a user; Sign language recognition means for recognizing the sign language of the user, sign language conversion means for converting the sign language of the user recognized by the sign language recognition means into natural language information, and natural language information converted by the sign language conversion means. Transmitting means for transmitting to the partner terminal.
[0010]
According to the mobile phone of the present invention, the sign language of the user photographed by the photographing means is recognized by the sign language recognizing means, and the recognized sign language of the user is converted into natural language information by the sign language conversion means. The natural language information converted by the means is transmitted to the partner terminal by the transmission means. Therefore, the intention of the hearing impaired person can be easily transmitted regardless of the distance to the opponent, and not limited to the hearing person or the hearing impaired person.
[0011]
Further, according to the mobile phone of the present invention, the natural language information converted by the sign language conversion unit is converted into a voice by the natural language voice conversion unit, and the natural language information is converted by the transmission unit by the natural language voice conversion unit. Since the voice corresponding to the sign language of the user is transmitted to the partner terminal, the intention of the hearing impaired person is easily transmitted regardless of the distance to the partner and not limited to the hearing person or the hearing impaired person. be able to.
[0012]
Further, according to the mobile phone of the present invention, the voice is received from the partner terminal by the receiving means, the received voice is converted into natural language information by the voice natural language information converting means, and the converted voice of the partner terminal is corresponded. Since the natural language information is displayed on the display means, the intention of the partner can be easily recognized regardless of the distance to the partner and even if the partner is a hearing person.
[0013]
Further, according to the mobile phone of the present invention, the natural language information corresponding to the voice of the partner terminal, which has been converted by the voice natural language information converting means, is converted into a sign language animation by the sign language animation converting means. Also, since the sign language animation corresponding to the voice of the partner terminal is displayed on the display means, it is possible to easily recognize the intention of the partner regardless of the distance to the partner and even if the partner is a hearing person. it can.
[0014]
Further, according to the mobile phone of the present invention, the natural language information corresponding to the sign language of the user, which is converted by the sign language conversion means, is displayed on the display means. You can check if it has been converted to natural language.
[0015]
Further, in order to solve the above-mentioned problem, the present invention relates to a communication method for performing wireless communication with a partner terminal using at least a voice and a character. Converting the sign language of the user into natural language information, transmitting the natural language information to the partner terminal, converting the voice received from the partner terminal into natural language information, and displaying the natural language information. I do.
[0016]
According to the communication method of the present invention, the natural language information obtained by converting the voice received from the partner terminal is converted into a sign language animation, and the sign language animation is displayed. Therefore, irrespective of the distance to the opponent, even if the opponent is a hearing person, the intention of the opponent can be easily recognized.
[0017]
In order to solve the above-mentioned problems, according to the present invention, in a communication program for performing wireless communication with a partner terminal using at least a voice and a character, a step of photographing a sign language of a user, Recognizing a user's sign language, converting the recognized user's sign language into natural language information, transmitting natural language information corresponding to the user's sign language to a partner terminal, and outputting voice from the partner terminal. The computer is made to execute a step of receiving, a step of converting the received voice into natural language information, and a step of displaying natural language information corresponding to the voice of the other party.
[0018]
According to the communication program of the present invention, a step of photographing the sign language of the user, a step of recognizing the photographed sign language of the user, a step of converting the recognized sign language of the user into natural language information, Transmitting natural language information corresponding to the sign language to the partner terminal; receiving voice from the partner terminal; converting the received voice into natural language information; and a natural language corresponding to the voice of the partner. And displaying the information. Therefore, the intention of the hearing impaired person can be easily transmitted regardless of the distance to the opponent, and not limited to the hearing person or the hearing impaired person.
[0019]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following, a configuration of a mobile phone possessed by a hearing-impaired person will be described. The hearing person may use a mobile phone for a hearing-impaired person described below to make a call, but this is not essential, and a normal (existing) mobile phone may be used. The ordinary mobile phone referred to here has at least voice input and voice output functions, and preferably has a display unit capable of displaying text characters.
[0020]
A. FIG. 1 is a block diagram showing a configuration of a mobile phone according to an embodiment of the present invention. In FIG. 1, a camera 1 is composed of an image sensor such as a CCD or a CMOS, and photographs an indoor person (in this case, a hearing-impaired person), and supplies the photographed image to an image recognition unit 2. The image recognition unit 2 recognizes the facial expression of the user, the movement of the arms, the shape and movement of hands (including five fingers), and the like, which are captured by the camera 1, and supplies the recognition result to the sign language recognition unit 3. The sign language recognition unit 3 refers to the standard sign language pattern dictionary 4 based on the image recognition result from the image recognition unit 2 and recognizes the sign language of the user. The standard sign language pattern dictionary 4 stores a sign language pattern in word units and a sign language animation in which the sign language pattern is animated.
[0021]
Incidentally, sign language used in Japan is roughly divided into "Japanese sign language" and "Japanese sign language". In Japanese-language sign language, sign language words are expressed in the order of Japanese words spoken as speech. On the other hand, Japanese Sign Language, which is used in conversations between hearing impaired persons whose native language is Sign Language, has a grammar different from that of ordinary Japanese. Mainly, words are expressed by finger movements, and much of the grammatical information is expressed by head movements, facial expressions, gestures, etc. other than finger movements. In the present embodiment, it is possible to determine whether the sign language is recognized by the sign language recognizing unit 3 and automatically switch any of the sign languages of “Japanese sign language” and “Japanese sign language”. Alternatively, the user may manually switch.
[0022]
The sign language / Japanese conversion unit 5 refers to the word dictionary 6 based on the result of the sign language recognition by the sign language recognition unit 3 and converts the sign language into a Japanese sentence by adding particles or adjectives. The word dictionary 6 stores a sign language pattern and Japanese in association with each other.
[0023]
Note that the sign language recognition algorithm and the sign language-Japanese conversion algorithm according to the present embodiment described above may be known, and for example, the technology described in Japanese Patent Application Laid-Open No. 6-67601 may be used.
[0024]
Next, the Japanese-to-speech conversion unit 7 recognizes the voice from the microphone 11 of the own terminal using a well-known voice recognition technology, and converts the voice into Japanese (text) with reference to the word dictionary 8. The Japanese-to-speech conversion unit 7 converts the Japanese sentence (text) by the sign language-to-Japanese conversion unit 5 into speech with reference to the word dictionary 8. The word dictionary 8 stores Japanese (text) and voice in association with each other.
[0025]
The display unit 9 includes a liquid crystal display, an EL display, a CRT, and the like, and includes a Japanese sentence corresponding to the sign language of the user of the own terminal and a Japanese sentence corresponding to the sign language of the other terminal (text: the other party is a hearing impaired person). , Japanese sentences corresponding to the voice of the partner terminal (text: the partner is a normal hearing person), sign language animation corresponding to the voice of the partner terminal (the partner is a normal hearing person), sign language image of the user of the partner terminal (the partner is a hearing impaired person) Is displayed as selectable. The speaker 10 outputs the voice of the user of the partner terminal (when the partner is a normal hearing person) or the voice from the Japanese-to-speech converter 7. The microphone 11 collects the voice of the user of the own terminal.
[0026]
The audio control unit 12 converts the audio data from the control unit 15 into an analog signal and supplies the analog data to the speaker 10, while digitizing the audio from the microphone 11 and supplies the digital signal to the control unit 15. Under the control of the control unit 15, the transmission / reception control unit 13 transmits a normal dial call, receives an incoming call, transmits a Japanese sentence (text) corresponding to the sign language recognized by the own terminal, and performs a Japanese-to-speech conversion unit 7. Transmission of voice corresponding to Japanese sentences, reception of voice from the partner terminal (the partner is a normal hearing person) and the like are performed. The key input unit 14 includes various function buttons, numeric keys for dialing and character input, and the like. The control unit 15 controls the entire operation of each unit described above.
[0027]
Next, FIG. 2 is a front view showing the appearance of the mobile phone according to the present embodiment. As shown in the drawing, a speaker 10, a camera 1, a display unit 9, a key input unit 14, and a microphone 11 are arranged in this order on the mobile phone. Note that the illustrated arrangement is an example, and various forms are possible, such as disposing the camera 1 on the back or folding the housing.
[0028]
B. Next, the operation of the mobile phone according to the present embodiment will be described in detail. With a mobile phone, a call is made to the other party's mobile phone, and when a line is established, transmission (sending) by a Japanese voice (or sentence) corresponding to voice from the own terminal or sign language and voice from the other terminal or Reception of Japanese voice (or sentences) corresponding to sign language is performed in full duplex.
[0029]
B-1. Transmission (transmission) operation First, the transmission (transmission) operation will be described. Here, FIG. 3 is a flowchart for explaining the transmission operation of the mobile phone according to the present embodiment. First, it is determined whether or not the user has dialed (Sa1). When the dial is made, a call is made by the transmission / reception control unit 13 to establish a line with the partner terminal (Sa2). Next, it is determined whether or not the call is a normal call (Sa3). In this mobile phone, there are a normal call mode in which a voice call is made and a sign language call mode in which a call is made in a sign language, and it is assumed that the user has previously selected which one to call.
[0030]
Here, when the normal call is selected, when the user inputs a voice from the microphone 11 (Sa4), the voice is converted into a Japanese sentence (text) by the Japanese-to-voice conversion unit 7 (Sa5). . Next, a Japanese sentence corresponding to the voice is displayed on the display unit 9 (Sa6). Then, the voice is transmitted to the partner terminal by the transmission / reception control unit 13 (Sa7). Next, it is determined whether or not the call has ended (Sa15). If the call has not ended, the process returns to step Sa3 and the above processing is repeated. On the other hand, when the call is terminated, the line is disconnected and the process is terminated (Sb16).
[0031]
On the other hand, if the sign language call is selected, the camera 1 shoots the user's sign language (including facial expressions, arm movements, finger shapes and movements) (Sa8). The image recognizing unit 3 recognizes the facial expression, arm movements, hand (including five fingers) shapes and movements of the other person photographed by the camera 1 (Sa9). Next, the sign language recognition unit 20 refers to the standard sign language pattern dictionary 4 based on the image recognition result by the image recognition unit 2 and recognizes the sign language of the user (Sa10). The sign language / Japanese conversion unit 5 refers to the word dictionary 6 based on the sign language recognition result of the sign language recognition unit 3 and converts the sign language of the user into a Japanese sentence (Sa11). The Japanese sentence corresponding to the sign language of the user is displayed on the display unit 9 of the terminal itself (Sa12).
[0032]
Next, the Japanese-to-speech converter 7 converts a Japanese sentence corresponding to the sign language of the user into a voice (Sa13). Then, a Japanese sentence (text) and / or voice corresponding to the sign language of the user is transmitted to the partner terminal by the transmission / reception control unit 13 (Sa7). Next, it is determined whether or not the call has ended (Sa15). If the call has not ended, the process returns to step Sa3 and the above processing is repeated. On the other hand, when the call is terminated, the line is disconnected and the process is terminated (Sb16).
[0033]
B-2. Reception (Reception) Operation Next, the reception (reception) operation will be described. Here, FIG. 4 is a flowchart for explaining the reception operation of the mobile phone according to the present embodiment. First, it is determined whether a character (text) has been received from the partner terminal (Sb1). Then, when the voice is received from the partner terminal, that is, when the partner terminal is a normal mobile phone (the partner is a normal hearing person), it is converted into a Japanese sentence by the Japanese-to-speech converter 7 (Sb2). Next, a Japanese sentence corresponding to the voice of the other party is converted into a sign language animation (Sb3). The Japanese sentence and / or the sign language animation corresponding to the voice of the other party are displayed on the display unit 9 (Sb4). Next, it is determined whether or not the call has ended (Sb5). If the call has not ended, the process returns to step Sb1 and repeats the above processing. On the other hand, when the call is terminated, the line is disconnected and the process is terminated (Sb6).
[0034]
On the other hand, if a Japanese sentence (text) is received from the partner terminal, that is, if the partner is a hearing-impaired person and talks in sign language using the same mobile phone as the terminal, the Japanese from the partner terminal is The sentence is directly converted into a sign language animation (Sb3). The Japanese sentence and / or the sign language animation is displayed on the display unit 9 (Sb4). Next, it is determined whether or not the call has ended (Sb5). If the call has not ended, the process returns to step Sb1 and repeats the above processing. On the other hand, when the call is terminated, the line is disconnected and the process is terminated (Sb6).
[0035]
Next, a call operation using the mobile phone according to the present embodiment will be described. Here, FIG. 5A is a conceptual diagram illustrating a call operation when the partner terminal is a normal terminal (hearing person), and FIG. It is a conceptual diagram which shows the call operation | movement when it is a hearing impaired person).
[0036]
First, when the partner terminal is a normal terminal, a voice is input to the partner terminal (Sc1), and the voice is transmitted to the terminal. The mobile phone receives a voice from the partner terminal (Sc2), converts the voice into a Japanese sentence (text) (Sc3), and further converts the Japanese sentence into a sign language animation (Sc4). The display unit of the mobile phone displays a Japanese sentence (text) or / and a sign language animation corresponding to the voice from the partner terminal. Therefore, the hearing-impaired person who is the user of the mobile phone can recognize that the other party is speaking by looking at the Japanese sentence (text) or / and the sign language animation displayed on the display unit.
[0037]
In addition, in the mobile phone, the user who is a hearing impaired will sign in the camera 1 (Sc5). The sign language is converted into a Japanese sentence (text) (Sc6), and the Japanese sentence is converted into a voice (Sc7). A Japanese sentence (text) or / and voice corresponding to the sign language is transmitted to the partner terminal. The other party terminal displays the Japanese sentence on the display unit when receiving the Japanese sentence (text), and outputs the sound from the speaker when receiving the sound (Sc8).
[0038]
Next, when the partner terminal is also the portable terminal (deaf person) according to the present embodiment, the partner who is a hearing impaired sign language toward the camera 1 (Sd1). The sign language is converted into a Japanese sentence (text) (Sd2). A Japanese sentence (text) corresponding to the sign language is transmitted to the mobile phone. The mobile terminal receives a Japanese sentence (text) corresponding to the sign language of the other party (Sd3) and converts the sentence into a sign language animation (Sd4). The display unit of the mobile phone displays a Japanese sentence (text) or / and a sign language animation corresponding to the sign language from the partner terminal. Therefore, the hearing-impaired person who is the user of the mobile phone can recognize the sign language of the other party by looking at the Japanese sentence (text) or / and the sign language animation displayed on the display unit.
[0039]
Also in the mobile phone, the user who is a hearing impaired will sign in the camera 1 (Sd5). The sign language is converted into a Japanese sentence (text) (Sd6). A Japanese sentence (text) corresponding to the sign language is transmitted to the partner terminal. At the partner terminal, a Japanese sentence (text) corresponding to the sign language of the user is received (Sd7) and converted into a sign language animation (Sd8). On the display unit of the partner terminal, a Japanese sentence (text) or / and a sign language animation corresponding to the sign language from the mobile phone are displayed. Therefore, the hearing-impaired person of the partner terminal can recognize the sign language of the user by looking at the Japanese sentence (text) or / and the sign language animation displayed on the display unit.
[0040]
Note that, in the above-described embodiment, a Japanese sentence (text) or / and a sign language animation corresponding to each other's sign language is displayed even in a call between hearing-impaired persons. This is because the bandwidth is narrow (the data transfer speed is low), so that it is difficult to transmit and receive a sign language image captured by a camera as a moving image.
[0041]
On the other hand, although not common yet, in recent years, there is a mobile phone that performs high-speed data communication using a high-speed line. In this case, moving images can be transmitted and received like a videophone. Therefore, if such high-speed data communication is possible, it is necessary to directly transmit and receive the sign language image of the hearing impaired taken by the camera, and convert it into Japanese text (text) or sign language animation. Is gone. Deaf people can directly see each other's sign language displayed on the display of the mobile phone.
[0042]
In the above-described embodiment, the telephone function as the remote communication device has been described. However, the present invention is not limited to this, and the present invention can be applied to a situation where the user is facing the other party. In this case, the user captures his / her own sign language, recognizes the sign language, converts the sign language into a Japanese sentence, and transmits it to the partner terminal. The voice of the other party is converted into Japanese text by voice recognition and displayed on the display unit. Even in such a face-to-face situation, by performing the conversion of own sign language and the voice of the other party using a mobile phone, it is possible to easily communicate with the other party, not limited to hearing-impaired or hearing-impaired persons. it can.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a configuration of a mobile phone according to an embodiment of the present invention.
FIG. 2 is a front view showing an appearance of the mobile phone.
FIG. 3 is a flowchart illustrating a transmission operation of the mobile phone.
FIG. 4 is a flowchart illustrating a receiving operation of the mobile phone.
FIG. 5 is a conceptual diagram for explaining a call operation using the mobile phone.
[Explanation of symbols]
Reference Signs List 1 camera (imaging means), 2 image recognition section, 3 sign language recognition section (sign language recognition means), 4 standard sign language pattern dictionary, 5 sign language / Japanese conversion section (sign language conversion means, sign language animation conversion means), 6 word dictionary , 7 Japanese ⇔ voice conversion unit (natural language voice conversion unit, voice natural language information conversion unit), 8 word dictionary, 9 display unit (display unit), 10 speaker, 11 microphone, 12 voice control unit, 13 transmission / reception control unit (Transmission means, reception means), 14 key input section, 15 control section

Claims (8)

In a mobile phone that bidirectionally transmits at least voice and text by radio communication with a partner terminal,
Photographing means for photographing the user;
Sign language recognition means for recognizing the sign language of the user photographed by the photographing means,
Sign language conversion means for converting the sign language of the user recognized by the sign language recognition means into natural language information,
Transmitting means for transmitting the natural language information converted by the sign language converting means to a partner terminal. Natural language speech conversion means for converting the natural language information converted by the sign language conversion means into speech,
2. The mobile phone according to claim 1, wherein the transmitting unit transmits the voice corresponding to the user's sign language converted by the natural language voice converting unit to a partner terminal. Receiving means for receiving audio from the other terminal;
Voice natural language information conversion means for converting the voice received by the receiving means into natural language information,
3. The mobile phone according to claim 1, further comprising display means for displaying natural language information corresponding to the voice of the partner terminal converted by the voice natural language information conversion means. Sign language animation conversion means for converting the natural language information corresponding to the voice of the partner terminal, which has been converted by the voice natural language information conversion means, into a sign language animation in which the sign language is animated,
4. The mobile phone according to claim 3, wherein the display unit displays the sign language animation corresponding to the voice of the partner terminal converted by the sign language animation conversion unit. The mobile phone according to any one of claims 1 to 4, wherein the display unit displays the natural language information corresponding to the sign language of the user converted by the sign language conversion unit. In a communication method for performing wireless communication with a partner terminal using at least voice and text,
Recognizing the photographed user's sign language, converting the recognized user's sign language into natural language information, transmitting the natural language information to the partner terminal,
A communication method comprising: converting speech received from the partner terminal into natural language information; and displaying the natural language information. 7. The communication method according to claim 6, wherein the natural language information obtained by converting the voice received from the partner terminal is converted into a sign language animation, and the sign language animation is displayed. In a communication program that performs wireless communication with a partner terminal using at least voice and text,
Capturing a sign language of the user;
Recognizing the sign language of the photographed user;
Converting the recognized user sign language into natural language information;
Transmitting natural language information corresponding to the sign language of the user to the partner terminal;
Receiving voice from the partner terminal;
Converting the received speech into natural language information;
Displaying the natural language information corresponding to the voice of the other party on a computer.

Images