JP2009152650A - Telephone device and speech communication translating method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve precision of data to be translated by a telephone device having a speech communication translating function. <P>SOLUTION: A telephone device includes a photography unit (101) which photographs a mouth expression of a speaker, a speech recognition unit (201) which converts speech data of the speaker into character data, an image analysis unit (205) which detects feature information associated with the mouth expression from image data, a DB (207) which stores relations between feature patterns associated with a plurality of mouth expressions and character data corresponding to the feature patterns, a retrieval unit (206) which retrieves a character data candidate corresponding to the feature information detected by the image analysis unit from the DB, a correction unit (202) which corrects speech recognition data according to a retrieval result, a translation unit (203) which translates the character data from the correction unit into character data of the language of a partner speaker, and a transmission unit (204) which generates transmission data to be sent out to a line using the translation data. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a telephone device, and more particularly to a telephone device having a function of translating a call between different languages.

  In order to facilitate the use of the telephone, the telephone device has various functions. One of them is the function of automatically translating and transmitting the speaker's voice. This function makes it easy to talk between users with different language types such as Japanese and English.

  For automatic translation of a call, the telephone device temporarily converts voice data input from the microphone of the transmitter to character data. The character data is subjected to a translation process corresponding to the language of the opposite speaker, and the translated data is sent to the line.

A cellular phone having the above translation function is described in, for example, Patent Documents 1 and 2 described later. The mobile phone described in Patent Document 1 translates and outputs voice input from a microphone and voice from a call partner into different languages. Patent Document 2 describes a technique for translating words by separating words so as not to disturb the rhythm of conversation when translating a call.
JP 2004-094721 A JP 2004-350298 A

  In the technology described in each of the above patent documents, data transmitted from a mobile phone is obtained by translating a speech recognition result for a speaker's utterance. However, depending on the content of the utterance and the voice quality of the speaker, an error may occur in speech recognition. When an error is included in the result of speech recognition, there arises a problem that even if the data is translated, data having a content different from the original content of the message is transmitted.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique for improving the accuracy of data to be translated in a telephone device having a call translation function.

  A telephone device according to the present invention relates to an imaging unit that captures a mouth expression of a speaker, a voice recognition unit that converts voice data of the speaker's utterance into character data, and the mouth expression from the image data of the mouth expression. An image analysis unit that detects feature information, a database that stores a relationship between a feature pattern related to a plurality of mouth expressions and character data corresponding to each feature pattern, and character data corresponding to feature information detected by the image analysis unit A search unit that searches the database for candidates, a correction unit that corrects character data from the speech recognition unit according to a search result of the search unit, and character data from the data correction unit in the language of the opposite speaker A translation unit that translates character data and a transmission unit that generates transmission data to be sent to a line using the character data from the translation unit are provided.

  The call translation method according to the present invention captures a mouth expression of a speaker who uses a telephone device, converts voice data of the speaker's utterance into character data, and relates to the mouth expression from the image data of the mouth expression. Information is detected, a candidate for character data corresponding to the detected feature information is searched from a database that stores a relationship between a feature pattern relating to a plurality of mouth expressions and character data corresponding to each feature pattern, and from the voice data The converted character data is corrected according to the search result, the corrected character data is translated into the character data of the language of the opposite speaker, and the transmission data to be sent to the line using the translated character data It is a method of generating.

  According to the present invention, it is possible to make call data to be translated more accurate. As a result, it is possible to prevent the content different from the content of the speaker's utterance from being transmitted to the opposite speaker, thereby enabling a smooth call between the speakers.

  FIG. 1 shows the configuration of an embodiment of the present invention. In the present embodiment, the telephone device according to the present invention is applied to a mobile phone 100. The mobile phone 100 includes a camera 101, a microphone 102, a speaker 103, an operation unit 104, a control unit 105, a radio unit 106, and an antenna 107. The operation unit 104 is provided with an LCD 104a for displaying a screen, a numeric keypad 104b and a switch 104c for operation.

  Camera 101 captures the mouth expression of the speaker and outputs image data of the mouth expression. The microphone 102 captures a speaker's utterance and inputs voice data of the utterance. The speaker 103 outputs the voice of the opposite speaker. The radio unit 106 transmits and receives radio signals for calls using the antenna 107. The control unit 105 controls the overall operation of the mobile phone 100.

  FIG. 2 shows a functional configuration of the control unit 105. The voice recognition unit 201 converts voice data input from the microphone 102 into character data by voice recognition processing. The image analysis unit 205 analyzes the image data input from the camera 101 regarding the mouth expression of the speaker, and detects feature information of the mouth expression. The image input by the camera 101 to the image analysis unit 205 captures the movement of the speaker's mouth by continuous shooting. The shooting range is not limited to the vicinity of the speaker's mouth, but may be the entire face. In the case of the entire face, the image analysis unit 205 discriminates the image portion around the mouth from the face image, and detects the feature of the mouth expression from the image portion. As the facial expression feature information, information indicating the relative position and the amount of change of the continuously taken mouth facial expression is used.

  A database (hereinafter referred to as “DB”) 207 registers the association between a feature pattern related to an arbitrary mouth expression and character data corresponding to the mouth expression represented by each feature pattern. At the time of registration, for example, one word or common phrase as character data is registered in association with one kind of feature of the mouth expression. The registration method may be any method such as transferring from an external terminal to the mobile phone 100 or downloading from an Internet site.

  The DB search unit 206 searches the character data in the DB 207 using the mouth facial feature information supplied from the image analysis unit 205 as a key. That is, a feature pattern that most closely approximates the feature information from the image analysis unit 205 is searched from the DB 207, and character data registered as a pair with the corresponding feature pattern is detected.

  The data correction unit 202 corrects the character data converted from the voice by the voice recognition unit 201 according to the search result of the DB search unit 206. In the correction, the data correction unit 202 compares the character data from the speech recognition unit 201 with the character data candidates from the DB search unit 206, that is, character data based on the mouth expression image. If there is a difference between the two, the character data from the speech recognition unit 201 is replaced with character data based on an image.

  The translation unit 203 converts the character data supplied from the data correction unit 202 into character data in the language of the opposite speaker. For example, when the language of the speaker of the mobile phone 100 is English and the language of the opposite speaker is Japanese, English character data from the data correction unit 202 is translated into Japanese. The user of the mobile phone 100 presets the language before and after translation at the start of a call or the like. In addition, any technique known in the technical field of the present invention can be applied to the translation processing and the translation dictionary acquisition.

  The transmission unit 204 prepares transmission data to be supplied to the radio unit 106 from the character data translated by the translation unit 203. As the transmission data, the translated character data can be converted into voice data, or the translated character data itself can be used. Which data format is used depends on the user's prior setting.

  The operation of the present embodiment will be described with reference to the flowchart shown in FIG. 3 and FIGS. 1 and 2. When the user starts a call on the mobile phone 100, the microphone 102 inputs the voice data of the speaker, and the camera 101 inputs the image data of the mouth expression (step S1). At this time, for the processing in the data correction unit 202 at the subsequent stage, for example, it is desirable to add information for synchronizing both to the corresponding audio data and image data. The synchronization information can be, for example, common time information.

  The voice recognition unit 201 converts voice data from the microphone 102 into character data (step S2). When recognizing image data continuously photographed by the camera 101, the image analysis unit 205 detects feature information of the mouth expression from them (step S3). The DB search unit 206 uses the feature information from the image analysis unit 205 as a search key, and searches the DB 207 for a feature pattern that is most similar to the search key. Then, a character data candidate paired with the corresponding feature pattern is detected (step S4).

  When the character data candidate is obtained from the DB 207 (step S5: Yes), the data correction unit 202 compares the candidate with the character data from the voice recognition unit 201 corresponding to the synchronization information (step S6). If the result of the comparison is that they do not match (step S7: No), the data correction unit 202 corrects the character data from the speech recognition unit 201 with the character data candidates from the DB search unit 206 (step S8). That is, the character data based on the voice is replaced with the character data based on the image.

  When no candidate is detected from the DB 207 (step S5: No), or when the character data from the speech recognition unit 201 matches the search candidate in the DB 207 (step S7: Yes), the data correction unit 202 The character data from the speech recognition unit 201 is supplied to the translation unit 203 without correction (step S9).

  The translation unit 203 translates the character data that has passed through the data correction unit 202 into character data in the language of the opposite speaker (step S10). The transmitting unit 204 confirms the preset setting related to the transmission data to the opposite speaker, and when transmitting the voice (step S11: voice), converts the character data from the translation unit 203 into the voice data and converts it to the radio unit. Supply to 106. When transmitting characters to the opposite speaker (step S11: characters), the character data from the translation unit 203 is supplied to the radio unit 106 as it is.

  The wireless unit 106 transmits voice data or character data as transmission data through the antenna 107 (step S13). Thereby, the utterance content of the speaker in the mobile phone 100 is transmitted to the opposite speaker in a corrected and translated state as necessary.

  A specific example of the above embodiment will be described with reference to FIG. In this example, a case is assumed in which the language of the speaker of the mobile phone 100 is English and the language of the opposite speaker is Japanese. Now, “It's ten fifty.” Is obtained as a character data string D1 based on speech, while “It's ten fifteen.” Is obtained as a character data string D2 based on an image. Suppose.

  In the above case, the data correction unit 202 recognizes that the character data “fifty” (d1) in D1 and the character data “fifteen” (d2) in D2 do not match, and converts the character data d1 based on speech. The character data d2 based on the image is replaced. With this correction, the character data string D3 “It ’s ten fifteen.” Is obtained. The translation unit 203 supplies the character data D4 “10:15” to the transmission unit 204 by translating the character data string D3 from English to Japanese.

  According to the present embodiment, the character data supplied to the translation unit 203 is corrected by the data correction unit 202, so that the character data to be translated can be made more accurate. As a result, content different from the content of the speaker's utterance is prevented from being transmitted to the opposite speaker, and smooth communication between the speakers becomes possible.

(Other embodiments)
In the above embodiment, when searching the DB 207, one candidate is detected. However, a plurality of candidates that approximate the search key within a predetermined range may be detected. The operation in this case will be described with reference to the flowchart of FIG. Here, the difference from the operation (FIG. 3) in the above-described embodiment will be mainly described.

  Similar to the above-described embodiment, the DB search unit 206 searches the DB 207 for the feature information of the mouth expression (step S4). As a result, when there is no search candidate (step S21: No), the character data from the speech recognition unit 201 is not corrected (FIG. 3: step S9). When one search candidate in the DB 207 is detected (Step S21: Yes, Step S22: No), the candidate is compared with the character data from the speech recognition unit 201 as in the above-described embodiment. (FIG. 3: Step S6).

  On the other hand, when a plurality of search candidates in the DB 207 are detected, the DB search unit 206 records the accuracy with respect to the search key for each candidate (step S23). This accuracy indicates the degree of approximation for the search key, that is, for the feature information from the image analysis unit 205. The numerical value of the accuracy can be obtained by comparing a feature amount such as a relative position or a change amount with respect to the mouth expression between the search key and the feature pattern.

  The data correction unit 202 compares each of the plurality of search candidates with the character data from the voice recognition unit 201. As a result, if there is a candidate that matches the voice recognition data (step S24: Yes), the data correction unit 202 supplies the voice recognition data to the translation unit 203 for translation as it is (FIG. 3: step S9).

  If none of the plurality of search candidates matches the voice recognition data (step S24: No), the data correction unit 202 refers to the accuracy of each candidate recorded earlier, and selects the candidate corresponding to the highest accuracy. select. Then, the voice recognition data is corrected by the selected candidate (step S25), and the corrected character data is supplied to the translation unit 203 (FIG. 3: step S10).

  FIG. 6 shows a specific example related to the above operation. In this example, it is assumed that three candidates “fifteen”, “fifty”, and “fifteenth” are detected from the DB 207, and the respective probabilities are “90%”, “70%”, and “40%”. If none of these candidates matches the speech recognition data, the data correction unit 202 selects “fifteen” corresponding to the highest accuracy “90%” for correction.

  On the other hand, when “fifty” of the candidates matches the speech recognition data, the speech recognition result (“fifty”) is translated without correction. Although “fifty” as a candidate is a candidate having a lower accuracy than the above “fifteen” in terms of image, priority is given to matching with the speech recognition result.

  As described above, according to the embodiment of FIG. 5, when a plurality of character data candidates based on the image are obtained and any one of the candidates matches the voice recognition data, the content of the voice recognition data is given priority. The When there is no candidate that matches the voice recognition data, the voice recognition data is corrected with the candidate having the highest accuracy. With this operation, speech recognition and image analysis can be adaptively reflected in character data to be translated. As a result, the accuracy of character data to be translated is further improved.

  Note that even if a certain candidate among the plurality of candidates matches the voice recognition data, if the accuracy of the candidate falls below a specified value, the voice recognition result may be corrected with the candidate having the highest accuracy. As a result, more accurate contents can be targeted for translation.

  The present invention is not limited to the mobile phone (100) having the wireless function as in the above embodiment, but can also be applied to a fixed telephone for wired communication. Further, in addition to a general call function, a telephone device having a videophone function for making a call while browsing each other's face images between speakers may be used. In the case of a videophone, the mouth expression of the speaker can be detected using the captured face image.

It is a block diagram which shows the structure of embodiment of this invention. It is a block diagram which shows the function structure of the control part in embodiment of this invention. It is a flowchart which shows operation | movement of embodiment of this invention. It is explanatory drawing regarding the specific example of embodiment of this invention. It is a flowchart which shows operation | movement of other embodiment of this invention. It is explanatory drawing regarding the specific example of other embodiment of this invention.

Explanation of symbols

100: mobile phone, 101: camera, 102: microphone, 103: speaker, 104: operation unit, 105: control unit, 106: wireless unit, 107: antenna
201: Voice recognition unit, 202: Data correction unit, 203: Translation unit, 204: Transmission unit, 205: Image analysis unit, 206: DB search unit, 207: DB

Claims (12)

A filming unit for photographing the mouth expression of the speaker;
A voice recognition unit for converting voice data of the utterance of the speaker into character data;
An image analysis unit for detecting feature information about the mouth expression from the image data of the mouth expression;
A database for storing the relationship between feature patterns related to a plurality of mouth expressions and character data corresponding to each feature pattern;
A search unit that searches the database for character data candidates corresponding to the feature information detected by the image analysis unit;
A correction unit that corrects character data from the voice recognition unit in accordance with a search result of the search unit;
A translation unit that translates the character data from the data correction unit into the character data of the language of the opposite speaker;
A telephone apparatus comprising: a transmission unit that generates transmission data to be transmitted to a line using character data from the translation unit.   The telephone device according to claim 1, wherein the correction unit replaces the character data with the candidate when the candidate from the search unit and the character data from the voice recognition unit do not match. When a plurality of candidates are detected from the database, the search unit records the accuracy of each candidate,
If any of the plurality of candidates does not match the character data from the voice recognition unit, the correction unit replaces the character data with a candidate with the highest probability of the plurality of candidates, and any one of the plurality of candidates 3. The telephone device according to claim 1, wherein when the character data matches the character data from the voice recognition unit, the character data is supplied to the translation unit.   4. The telephone device according to claim 1, wherein the transmission unit generates voice data as the transmission data from character data from the translation unit. 5.   The telephone device according to any one of claims 1 to 3, wherein the transmission unit outputs character data from the translation unit as the transmission data.   6. The telephone device according to claim 1, further comprising a wireless unit that transmits the transmission data to a line by wireless communication. Take a picture of the mouth of the speaker using the phone
Converting voice data of the utterance of the speaker into character data;
Detecting feature information related to the mouth expression from the image data of the mouth expression;
Search for candidate character data corresponding to the detected feature information from a database that stores the association between the feature patterns related to a plurality of mouth expressions and the character data corresponding to each feature pattern,
Correct the character data converted from the voice data according to the search result,
Translating the corrected character data into the character data of the language of the opposite speaker;
A call translation method characterized by generating transmission data to be sent to a line using the translated character data. In correction of character data converted from the voice data,
8. The call translation method according to claim 7, wherein if the candidate as a result of the search and the character data converted from the voice data do not match, the character data is replaced with the candidate. When a plurality of candidates are detected from the database as a result of the search, the accuracy of each candidate is recorded,
In the correction of the character data converted from the speech data, if none of the plurality of candidates matches the character data converted from the speech data, the character data is replaced with the candidate with the highest probability of the plurality of candidates. The call translation according to claim 7 or 8, wherein if any of the plurality of candidates matches the character data converted from the voice data, the character data is translated as the corrected character data. Method.   10. The call translation method according to claim 7, wherein voice data is generated from the translated character data as the transmission data.   The call translation method according to claim 7, wherein the translated character data is output as the transmission data.   12. The call translation method according to claim 7, wherein the transmission data is transmitted to a line by wireless communication.

Images