KR101445904B1 - System and methods for maintaining speech-to-speech translation in the field - Google Patents

Abstract

There is provided a method and apparatus for updating a vocabulary of a speech translation system that translates a first language into a second language including an octopus and a spoken language. The method includes adding a new word in the first language to a first recognition vocabulary of the first language and associating a predetermined description including pronunciation and word class information with the new word. The new word and the comment are then updated in a first machine translation module associated with the first language. The first machine translation module is configured to include a first tagging module, a first translation model and a first language module and translate the new word into a corresponding translation word in the second language. Alternatively, the present invention may be used for unidirectional or multi-directional translation.

Description

[0001] SYSTEM AND METHOD FOR MAINTAINING SPEECH TRANSLATION IN THE FIELD [0002]

The present invention relates generally to speech translation systems for interlanguage communication, and more particularly to a system and method for enhancing and improving the contents and usage of a system by adding new vocabulary items in the field without requiring the user to have linguistic or technical knowledge or expertise. And more particularly to a method and apparatus for on-site maintenance.

Mutual citation of related application

This application is a continuation-in-part of U.S. Provisional Patent Application No. 61 / 045,079 filed on April 15, 2008, provisional U.S. Provisional Patent Application No. 61 / 092,581 filed on August 28, 2008, 093,898 (filed on September 3, 2008).

Automatic Speech Recognition (ASR) and Machine Translation (MT) technologies have developed to the point where it is possible to develop real speech translation systems in laptops and mobile devices in limited and unlimited domains. In particular, region-limited speech translation systems have been developed at research sites and laboratories in a variety of applications, including tourism, medical deployment and military. Such a system is called " A. Waibel, C. Fugen, "Spoken language translation" in Signal Processing Magazine, IEEE May 2008; 25 (3): 70-79, In Proc. HLT, 2003 "; &Quot; The CMU TransTac 2007 eyes-free " in the book by Nguyen Bach, Matthias Eck, Paisarn Charoenpornsawat, Thilo Kohler, Sebastian Stuker, Thuy Linh Nguyen, Roger Hsiao, Alex Waibel, Stephan Vogel, Tanja Schultz and Alan W. Black and a hands-free two-way speech-to-speech translation system. However, these systems are limited in that they operate with a finite set of vocabularies that are predefined by the system developer and dictated by the application area and location that the system is supposed to be used. Therefore, most vocabulary and language usage is based on example scenarios and is determined by data collected or assumed in that scenario.

However, in the field, actual words and language usage deviate from the expected scenarios of the laboratory. Even in simple areas such as tourism languages, usage in the field will vary greatly as users travel to different places, communicate with others, and pursue different purposes and needs. Therefore, new words and new expressions will always be created. In this new word-speech recognition term, an out-of-vocabulary (OOV) word is mistakenly recognized as a word in the vocabulary dictionary and can be incorrectly translated. The user may be able to express in other words, but if an important word or concept (such as a person's name or a city's name) can be entered or communicated, it may not be communicated because there is no word or expression.

A user-changeable voice translation system is needed, but no real solution has been presented so far. Adding words to the system seems easy, but it turns out that such a change is very difficult. Appropriate changes to many component modules have to be made throughout the system and most modules have to be re-learned to restore the balance and integration of the components. In fact, to learn a new word, about 20 different modules had to be changed or re-optimized. This change requires expertise and experience with the components of the voice translation system, and as such, the inventors have known that such changes have been made by experts in the laboratory so far, and are expensive and time-consuming.

For example, if a system designed for users in Europe does not include the name "Hong Kong" in the vocabulary dictionary, the system will recognize the word in the dictionary closest to the pronunciation if the speaker says "Let's go to Hong Kong" To "Let's go to home call". It is not clear at this time whether the error is due to a recognition error or the word is not in the speech translation system. Therefore, the user corrects the system. This work can be done in accordance with one of several correction techniques. The simplest may be rephrasing or typing, but it may be more effective to use cross-modal error correction techniques as described in other documents and prior art (US Pat. No. 5,855,000 to Waibel et al.). Once the correct spelling of the desired word sequence is established ("Let's go to Hong Kong"), the system performs translation. If there is "Hong Kong" in advance, the system will proceed from there to work normally and perform translation and synthesis. However, if the word is not in the recognition and translation dictionary, the system will need to determine whether it is a named entity or not. Finally, and most importantly, even if names or words can be correctly translated into the output language through user intervention without learning, the system may not recognize and translate it correctly the next time the user speaks the word.

Unfortunately learning a new word is not a problem that can be solved by typing a new word into a vocabulary, but rather about 20 different points and a whole level of the speech translation system. For now, this is done to re-establish consistency between all elements and the dictionary of elements and to restore the statistical balance between words, phrases, and concepts in the system (the probability should be 1, Tagging and editing entries, collecting a wide range of databases containing the required words, re-learning the language model and translation model probabilities, and re-optimizing the overall system.

As a result, existing voice translation systems would have to make use of advanced computing tools and language resources, usually in the lab, to make very few changes. However, it does not seem possible in terms of time, effort and cost to make all the changes in the laboratory for actual field use. Instead, the user can perform all important operations and language processing steps in a semi-autonomous or autonomous manner without complexity, and communicate with the user as simple and clear as possible through a simple intuitive interface. Learning and custom modules that do not require expert knowledge are required. The present invention provides in detail learning and user-customized modules that can meet these needs.

Unfortunately, translation systems are usually so complex that user access is not really possible. Therefore, a system and a method that enable users to easily change even if the user does not have expertise in language or technology by using machine translation technology, to enable multilingual communication, to overcome language barriers and to make people closer Is required.

SUMMARY OF THE INVENTION [

In various embodiments, the present invention solves these problems by providing a method and apparatus for updating a vocabulary of a speech translation system. In various embodiments, a method is provided for updating a vocabulary of a speech translation system that translates a first language into a second language comprising an octopus and a spoken word. The method includes adding a new word in the first language to a first recognition vocabulary of the first language and associating a predetermined description including pronunciation and word class information with the new word. The new word and the comment are then updated in a first machine translation module associated with the first language. The first machine translation module is configured to include a first tagging module, a first translation model and a first language module and translate the new word into a corresponding translation word in the second language.

Optionally, for bidirectional translation, the method further comprises translating the translation word back into the new language in the second language, correlating the new word with a corresponding translation word in the second language, And adding the translated word and the comment to the second recognized lexicon of the second language. The second machine translation module associated with the second language is then updated with the translation word and the comment. The second machine translation module includes a second tagging module, a second translation model, and a second language module.

In embodiments, the method further comprises inputting a first word into a text-to-speech pronunciation lexicon associated with the first language and inputting a second word into a text-to-speech pronunciation lexicon associated with the second language . The input signal may be in a different form (e.g., speech and non-verbal spelling, speech and verbal spelling, text and speech) (referred to herein as "cross-modal" Writing, and repeated writing).

An embodiment of the invention relates to a field-maintainable class-based speech translation system that communicates between a first language and a second language. The system comprises two speech recognition units each adapted to receive a sound comprising a spoken language of a first or a second language and to generate text corresponding to the spoken word and to receive text from one of the speech recognition units And two corresponding machine translation units configured to output the translation of the text into another English text. The system also includes a user field customization module that allows the system to work with the user to learn new words. The user field customization module is configured to accept user selection inputs that include sound or text corresponding to one or both of the languages, and suitably update the machine translation units with the user selection input.

In one embodiment, the system has four key features that enable it to be a field-maintainable class-based speech translation system. First, it includes a speech translation system that enables the addition of new words to active system vocabularies and the replacement of position or task specific vocabularies. This makes it possible to add dynamic words to the speech recognition module without having to restart the module. The system is based on a combination of multilingual system dictionaries and language-independent word classes, class-based machine translation (phrase-based statistical MT, syntactic, example-based, etc.) Use multilingual word class tagging during model learning, and word class tagging in a new language through alignment through parallel corpus from a known tagged language. Second, a multimodal interactive interface allows non-experts to add new words to the system. Third, the system is designed to accommodate ASR and SMT model adaptation using multi-modal feedback provided by the user. Fourth, the system has a networking function that can correct or share words.

In another embodiment, a multimodal interactive interface is disclosed that allows a user to add new words to a speech translation device in the field without technical expertise. Examples include (1) automatically classifying words or phrases to be added to the system and automatically generating pronunciation and translation of the words; (2) a method of inputting new words in a cross-modal manner with at least one of speaking, typing, spelling, handwriting, browsing and paraphrasing; (3) Multimodal feedback to help a linguistically untrained user determine whether phonetic transcriptions and translations are appropriate: Multiple text forms through text-to-speech conversion (TTS: this sounds appropriate) In Roman form, as well as script form in script form) and voice form; (4) a method for setting a language model and translation probability for a new word; And (5) boosting or decreasing the language model and translation probability for the newly learned word based on relevance to user activity, interest, and usage history.

In another embodiment, an on-line system is disclosed that corrects on-site with multi-modal user feedback. Examples include (1) an interface and method that allows a user to correct an automatic speech recognition result and adapt the speech recognition module using the feedback information; (2) an interface and method that allows a user to correct machine translation sample statements and use this feedback information to enhance machine translation components; And (3) automatically adjusting (increasing or decreasing) the language model, dictionary, and translation model probabilities for correct words or corrected words based on user corrections.

In another embodiment, an Internet application is disclosed that allows a user to share a correction or new word addition in the field across multiple devices. By way of example, (1) a method for uploading, downloading and editing models used in a voice translation device over the World Wide Web; (2) a method of collating the addition and correction of new words on-site across the user's entire community; And (3) uploading, downloading and editing location-specific or business-specific vocabularies used in the speech translation device.

The accompanying drawings illustrate examples of embodiments of the present invention. In the drawings,
1 is a block diagram illustrating a speech translation system constructed in accordance with an embodiment of the present invention.
Figure 2 illustrates an example of a graphical user interface displayed to a user via a tablet interface;
Figure 3 is a flow chart showing the steps of voice translation performed in accordance with the embodiment of the invention of Figure 1;
4 is a flowchart showing steps (correction and calibration module) in which the system learns from correction by a user.
5 is a flow chart showing the steps by which a user can add a new word to the system (a user field customization module);
6 is a flow chart showing an example of a method by which a device automatically generates a translation and pronunciation of a new word that a user wishes to add to the system.
7 is a flow chart illustrating an example of a method for verifying new word input through a multimodal interface;
8 is a diagram showing an example of a visual interface for displaying automatically generated word information;
Figure 9 is a flow chart showing the steps required to learn a class-based MT model.
10 is a flow chart showing the step of applying a class-based MT to an input sentence.
Figure 11 is a diagram showing possible features used during word class tagging through statistical or mechanical learning schemes.

Various embodiments of the present invention describe a method and system for speech translation. Embodiments can be used to adapt to the user ' s voice and speech through model application. In a further embodiment, the user can correct the recognition error, and the system can learn clearly from the error corrected by the user, so that these errors are less likely to occur again later. In accordance with the present invention, the user can tailor the vocabulary to his or her personal needs and environment by adding new words to the system or selecting a predetermined dictionary optimized for a particular location or task. When adding a new word, the user can correct and verify the automatically generated translation and pronunciation using a multimodal interface. Accordingly, the user can add a new word to the system without knowing another language. In an embodiment, the system is further configured to communicate a new vocabulary entered by the user to a user community. This data is verified, a dictionary is automatically generated and any user can download the dictionary.

1 is a block diagram illustrating an example of a field-maintainable speech translation system in accordance with an embodiment of the present invention. In this example, the system operates between two languages, L _a and L _b . This is _a common implementation of a speech dialogue system in both directions, namely from L _a to L _b and from L _b to L _a , with respect to speech translation. However, the bi-directionality of this configuration is not essential to the present invention. A unidirectional system from L _a to L _b or a multi-directional system with respect to several languages L ₁ , ..., L _n can likewise benefit from the invention. The system recognizes the speech for each of L _a and L _b and generates a corresponding acoustic model 18, an ASR class-based language model 19 and a lexicon model 20, 3) for generating text corresponding to L _a and L _b using a plurality of ASR modules (2, 9). This example uses the "Ninja" speech recognition system developed by Mobile Technologies, LLC. Other types of ASRs available are IBM, SRI, BBN, or speech recognizers developed by Cambridge or Aachen.

The system also includes two machine translation modules (3, 8), each of which translates text from L _a to L _b and from L _b to L _a . The MT module used in this example is a "Pandora" system developed by Mobile Technologies, LLC. Other MTs available include those developed by IBM, SRI, BBN, or Aachen University.

The two text-to-speech engines 4 and 7, respectively corresponding to the machine translation modules 3 and 8, are configured to receive the text generated in the corresponding ASR unit. The output text is sent to its own MT module (3 or 8), each of which translates the text from L _a to L _b and from L _b to L _a . The TTS module generates an audio output for converting at least one text word in L _a to speech through an output device 5 such as a loudspeaker and an audio output for converting at least one text word in L _b to speech Such as the output device 5 or the loudspeaker 6. In this example, a Cepstral TTS module is used. TTS modules that support the Windows speech application programming interface (SAPI) can also be used.

The correction and calibration module 11 allows the user to correct system output through various forms including voice, gesture, writing, touch, touch-sensitive and keyboard interface, and allows the system to learn from user corrections I will. This correction and calibration module may be in the form as disclosed in U.S. Patent No. 5,855,000. The user field customization module 12 provides the user with an interface to add a new vocabulary to the system and may select an appropriate system vocabulary for the user's current situation. For example, this may be triggered according to a change in the location found by the GPS coordinates indicating the current location of the device or according to an explicit selection of the user's job or location.

The user accesses the user field customization module 12 to interact with the system through a graphical user interface displayed on the screen of the device 13 (or an active touch screen) and a pointing device 14 including a mouse or pen . An example of a graphical user interface is shown in FIG. In this example, the device 13 displays in the window 15 the text of the audio input of L _a and its corresponding text. Text machine translation to a second language of L _a L _b are displayed in the window 16.

In the embodiment, the same microphone and loudspeaker can be used in both languages. Thus, the microphones 1, 10 may be a single physical device, and the speakers 5, 6 may also be a single physical device.

A flowchart illustrating the operation of an example of the method of the present invention is shown in Fig. First, in step 15b, the speech recognition system is operated by the user. For example, a button on a graphical user interface (item 15b in Fig. 2) or an external physical button (not shown) may be selected. Then the module 2 by the ASR module at step 27, i.e. if the user speaks L _a , or by the module 9 if the user speaks L _b , Item 25) is recognized. The ASR modules 2 and 9 apply three modules: an acoustic model 18, an ASR class-based language model 19, and a recognition vocabulary model 20. These models are language specific and each ASR module contains its own set of models. At step 28, the final text of the user's voice is displayed via the GUI of the device screen 13.

The translation is then performed via the MT module (3 or 8) according to the input language. The MT modules 3 and 8 include three main models: a tagging or parsing [Collins 02] model (model 22), a class-based translation model (model 23) A class-based language model (model 24) is applied. The tagging model 22 is described in " J. Lafferty, A. McCallum, and F. Pereira, "Conditional random fields: Probabilistic models for segmenting and labeling sequence data," In Proceedings of 18th International Conference on Machine Learning, pages 282-289, 2001 ("Lefferty01" Such as the form described by Michael Collins, " Parameter estimation for statistical parsing models: Theory and practice of distribution-free methods "(2004) In Harry Bunt, John Carroll, and Giorgio Satta, New Developments in Parsing Technology, Kluwer Type tagging or parsing model. Other models applied during machine translation include the distortion model and the sentence length model, which limit the word rearrangement method during translation. Details of class-based machine translation are described below. The final translation is displayed via the GUI of the device 13 as shown in step 30.

In order to assist the user in determining whether the translation output is appropriate, an automatically generated translation (Figure 2, item 16) is re-translated into the input language via the MT module 3 or 8, for example as shown in item 15a of Figure 2 It is displayed in parentheses under the circle input. If the reliability of both speech recognition and translation is high as determined by the ASR model (2 or 9) and the MT model (3 or 8) (step 31) And is generated via the loudspeaker 5 or 6 (step 33). Otherwise, the system will display via GUI, audio and / or tactile feedback that translation may be incorrect. The particular TTS module used in step 33 is selected according to the output language.

Thereafter, if the user is not satisfied with the generated translation, the user may intervene either during the speech translation process at steps 27-33 or after the process is completed. This activates the correction and calibration module 11 (step 35). The correction and calibration module 11 records and logs the corrections that can be made by the user and these corrections can be used later to update the ASR modules 2 and 9 and the MT modules 3 and 8. Which will be described in more detail later in this specification. If the correction includes a new lexical item (step 36), the user enters a field customization mode that explicitly adds a new word to the system at step 15c, or at step 15d the "Thomas Schaaf, Detection of OOV words using generalized word models and a semantic class language model, "in Proc. of Eurospeech, 2001 ", a new field is automatically detected in the input audio using reliability or a new word model, the user field adaptation module 12 is triggered. This module 12 provides a multimodal interface that allows the user to add new words to active system vocabularies. If the user adds a new word or phrase, the ASR, MT and TTS models (items 17, 21 and 33a) are updated as needed. The functionality of this module is described in more detail later in both languages.

For both languages, a generic set of classes (e.g., name, place name, and organization name) is used in both ASR and MT. It provides a system-wide set of semantic slots that allow new words to be added to the system. Names, jargon, and expressions that occur within these classes are highly variable words that are based on different user placements, locations, cultures, customs, and tasks, so they are highly customizable.

The specific class used in the preferred example depends on the application of the system. A class is a named-entity; The names of people, places and organizations; Or task specific noun phrases; A semantic class for a food, a disease or drug name; And other public classes for words or phrases that do not fit into a predetermined class. Syntactic classes such as synonyms or word equivalence classes may also be used. Examples of applications include, but are not limited to, tourism, medical care, and peace keeping. As an example, classes required in the field of tourism application include names of people, cities, and food names. As another example, the classes required in the medical profession include disease names, drug names, anatomical names, and the like. As another example, the classes needed for peacekeeping include the bearer name, vehicle name, and so on. To enable field-specific voice translation, the system can perform error correction through the operation of the correction and calibration module 11 along with the user field customization module 12, and then learn from these errors.

Correction and Calibration Module

The correction and correction module 11 allows the user to intervene in the voice translation process at any time. The user can identify and log errors, or correct errors in speech recognition or translation output, if desired. Such user intervention is very valuable because it can immediately correct errors in the process of communication between people and the system provides opportunities to learn from mistakes, tailored to user needs and interests. A flow chart illustrating such an error feedback function is shown in Fig. If the user does not like the translation of a word (i.e., an error occurs), the user can record the current input (step 40). The system will store the audio as well as other information in the log file. This log file can be accessed and corrected later by the user and uploaded to the common database so that the professional user can identify and correct the error.

The user can also correct speech recognition or machine translation output through various modalities. The user can correct the entire word by repeating the whole word or typing the sentence through the keyboard or handwriting interface. Alternatively, the user may highlight an error portion of the output hypothesis through a touch screen, a mouse or a cursor key, using only the phrase or word using keyboard, handwriting, or voice or writing the word one character at a time Can be corrected. The user can select an error portion of the outputted sample sentence through the touch screen and select a sample sentence that is considered to be correct among the automatically generated drop-down list, or to select the error portion as a voice or another supplementary form For example, handwriting, spelling, paraphrasing). ≪ / RTI > A suitable combination of these methods and supplemental calibration work is based on the method described in U.S. Patent No. 5,855,000 to Waible et al. For multimodal speech recognition correction and correction. Here, these methods are applied to the speech recognition and translation module of an interactive speech translation system.

If the user corrects the speech recognition output (step 43), the system first determines whether the correction includes the new word (step 44). This determination is made by checking the words in the recognition lexicon model 20 associated with each language La and Lb. If the word can not be found, the system prompts the user to add the new word to the active system vocabulary (Fig. 5, step 50). Otherwise, the probability in the ASR model (FIG. 3, item 17) is updated to reduce the likelihood of the same error occurring again. This can be done in a discriminative way in which the probability of the corrected word sequence is increased and the probability of a close-competing sample query is reduced.

The user may correct the machine translation output if the linguistic knowledge is sufficient. The same format as used in the ASR case can be used. If the machine translation output is corrected by the user (step 45) and the correction contains a new word, then a dialog is presented to the user to add the new word to the active system vocabulary (Figure 5, step 50) ). If this correction already includes only words in the active system, the machine translation module (item 3, item 21) is updated. In particular, implementations may be used in which phrases are extracted from corrected sentence pairs and mixed in translation modules. The target language model used can be updated in a similar manner to the ASR case.

User field customization module

The user field customization module 12 allows the system to learn new words in cooperation with the user. In the conventional system, the user can not change the vocabularies in the voice translation system. Unlike the conventional system, the user field customization model 12 enables the user to change vocabularies relatively easily and gradually as a non-expert, without knowing computer speech and language processing techniques or linguistics in the execution system. The model 12 provides and accepts easily understandable feedback from the user and autonomously derives both necessary parameters and system configurations based on the feedback to provide such field alignment. The field customization module 12 alleviates the burden on the user by accomplishing this through 1) an intuitive interface for user customization and 2) an internal tool that dynamically evaluates both internal parameters and settings needed for customization give.

In one-way translation, the system processes at least four pieces of information about the word or phrase to add new words or phrases to the active system vocabulary. This information includes:

● Class (ie the semantic or syntactic class of the new entry)

● Words in the language L _a (ie documents in L _a )

● Pronunciation of a word with L _a

● Translating a word into L _b (ie a document with L _b )

In bi-directional translation, the system also requires input of pronunciation of a new word in L _b . L _b allows the TTS to generate an audio output and allows the ASR module for L _b to recognize the new word in reverse.

The flowchart illustrating the operational steps of the user field customized model 12 is shown, for example, in FIG. When a new word is entered into the system, based on the correction and calibration intervention in the previous section, the system will inform the user whether the word should be "learned ", i.e. added to the active system vocabulary (Fig. 5, step 50). If so, the word learning mode is activated and the field alignment module 12 begins to operate. Note that field fit or new word learning does not only require results from the error correction dialog. The user may explicitly choose to enter a word learning mode from the pull down menu and add a new word or a new word list a priori. New word learning can be activated even when suddenly many words are needed, such as special terms, names, and locations. In all such cases, however, the system must collect the above information.

If the user notices that he or she wants to add a new word to the system vocabulary (step 50), the system first accesses a dictionary containing the dictionary itself, which is either contained in the device itself or accessible via the Internet, Search. This external dictionary consists of entries of word translation pairs. Each entry contains not only pronunciation, but also word class information that allows new words to be easily added to the active system vocabulary. Each entry also contains a description of each word pair in both languages. Accordingly, the user can select a proper translation of the word even if the user does not have knowledge of the target language. If a new word is included in the external dictionary (step 51), the system displays the alternate translation list of the word with the description of each of the translations (step 52). If the user selects one of the predefined translations from the dictionary (step 53), the user can verify the pronunciation and other information provided by the dictionary (step 53a), and if necessary, edit You may. This new word is then added to the active system vocabulary.

Three steps (steps 59, 59a, 59b) are required to add a new word to the active system vocabulary. First, the word and its translation are added to the ASR-aware vocabulary of module 2, 9 (step 59). This word is added to this recognition lexicon 20 along with the pronunciation given in the dictionary. As the user enters the word, the occurrence probability is set to be larger than the contention word of the same class in the ASR class-based language model 19. [ This is to ensure that the probability that a word added by the user will occur is greater. Next, the word and its translation are added to the MT model (FIG. 3, item 21) and the system can then translate the new word into both. Finally, the word is registered in the TTS pronunciation model (FIG. 3, model 33a), which allows the system to pronounce the word correctly in both languages.

If this new word entered by the user is not in the external dictionary, the system will automatically generate the information necessary to register the word in the active system vocabulary and the user will verify this information. First, the class of this new word is evaluated (step 54) through a tagging model (FIG. 3, model 22) using the surrounding word context (if available). Next, pronunciation and translation of this new word is automatically generated (step 55) via a rule-based or statistical model. The generated information is then shown to the user via the multimodal interface (step 58). The system verifies (step 58) and corrects (step 57) the automatically generated translation or pronunciation to the user. Finally, after the user has verified this information, this new word is added to the active system vocabulary (steps 59, 59a, 59b). In order to dynamically add a new word (specifically "word + pronunciation + word class") to the ASR vocabulary 59, a recognition vocabulary (typically stored in a tree structure in the ASR module 2 or 9) (20) is retrieved and then updated to include the new word. Thus, the new word can be added dynamically to the recognition vocabulary, so that it can be immediately recognized when it is said in the following words. The ASR system does not need to reinitialize or restart as in conventional systems.

Similarly, a new word (specifically "word + pronunciation + word class") may be attached to the MT translation model 59a and stored as a hash-map within MT module 3 and / Translation model 23) is searched and a new translation pair including this new word, its translation and the word class is attached. This new word can thus be dynamically added to the MT model (3 and / or 8) and will be correctly translated in the proceeding speech. The MT system does not need to be reinitialized or restarted as in conventional systems.

Even non-specialists in this field are required to automatically evaluate all this information so that they can be customized. The following sections describe how to automatically assess this important intermission of words and how they can be obtained or verified intuitively by the user.

Pronunciation and translation of new words

Users of a speech translation system usually do not know about phonetics, linguistics, or language skills, nor do they know the words and their usage in other languages. Therefore, the translation of new words to be added to the system and all related information , Word usage, etc.) can not be expected. Therefore, when the user enters a new word, the system evaluates the word class and automatically generates the translation and pronunciation information of the word in both languages.

To add a new word to an active system vocabulary, it is necessary to translate the word and pronounce the word and its translation. This information may be generated in accordance with, for example, a three-step process shown in FIG. First, pronunciation of a word is generated (step 60). A translation is generated based on the character sequence of the word and its pronunciation (step 61). Next, pronunciation of a new word in the target language is generated based on the information generated in the previous step (step 62). Two examples of generating this information using various techniques within the S2S translation system that can be maintained in the field of the day are shown on the left side of FIG. To add a new English word "Wheeling " to the system (item 61), the English pronunciation is first generated through machine learning (step 61). Machine learning is described in " Damper, R. I. (Ed.), Data-Driven Techniques in Speech Synthesis. Dordrecht, The Netherlands: Kluwer Academic Publishers (2001). &Quot; Next, a translation of the word into Japanese is automatically generated (step 66) through statistical machine translation, and then Japanese pronunciation is generated in accordance with the passive definition rule (step 67). Translation can be accomplished using any suitable statistical machine translation engine. This example is based on the "K. Knight and J. Graehl, Machine transliteration. Computational Linguistics 24 4 91998), pp. 599-612 " and Bing Zhao, Nguyen Bach, Ian Lane, and Stephan Vogel, "A Log-linear Block Transliteration Model based Bi-Stream HMMs ". The generated information (item 68) is then verified by the user via the acoustic drive and the voice sequence before registering the word in the active system vocabulary.

Similarly, in order to add a new Japanese word "Wakayama" (item 70) to the system, Japanese pronunciation is first generated in accordance with the manual definition rule (step 71). Next, the translation of this word in Japanese is automatically generated in accordance with the rule base translation (step 72), and then the English pronunciation is generated via the passive definition rule (step 73). This rule base translation is described in Mansur Arbabi, Scott M. Fischthal, Vincent C. Cheng, and Elizabeth Bar, "Algorithms for Arabic name transliteration," IBM Journal of research and development, 38 (2): 183-193, 1994 Method. ≪ / RTI > The generated information (item 74) is then verified by the user before registering the word in the active system vocabulary.

The user can verify the generated translation and pronunciation by audible output. Alternatively, if the user thinks it is more appropriate, the document may be in a native language (i.e., "Hanyu Pinyin" in Chinese if the user is an English user, or "Romaji" in Japanese). The user can edit the translation and / or pronunciation as needed. Once approved by the user, the word and word characteristics are added to the multilingual system dictionary.

The system also automatically generates the necessary information according to the interactive user input, thereby eliminating the need to translate new words that are added in advance. An example of a user interface is shown in FIG.

Interactive user interface

The system then allows the user to verify and verify the evaluated language information. This is done intuitively so as not to assume any particular linguistic or technical knowledge. Therefore, a suitable interface is used. The following describes user interaction during learning of new words.

At the interface, the user can select the "new word" mode from the menu, or the new word learning mode can be triggered after the user correction has generated a new / unknown word. Through the window that displays, the user can now type new words, names, jargon, concepts, and expressions that they want. Based on the spelling input in the user's language (which can be a non-English character set, such as Chinese, Japanese, Russian, etc.), the system generates the Roman alphabet and the expected word pronunciation. This is done according to the conversion rules learned from handwriting conversion rules or from voice data extracted or translated from existing speech dictionaries. Then, the user can view the automatic conversion and drive the sound of the generated pronunciation through the TTS. The user can iterate and change any of these expressions (script in each language, translation in romanization, phonetic transcription and the sound), and other corresponding entries can be re-generated as well (thus, The notation can change the notation in another language).

The system can also automatically select the most likely word class to which the new word belongs based on the coincidence statistics of other words (with known classes) in a similar sentence context. However, it is also possible to manually select (and / or correct) this class identity through a new word window so that the user can ignore such evaluated class evaluation.

In summary, given a new word / phrase from the user,

Automatically classify the semantic classes of entries (used by ASR and MT components)

• Automatically generate pronunciations for words (used by ASR and TTS for L ₁ )

● Automatically generates a translation of the word (used by all MT components)

• Automatically generate pronunciations for words (used by ASR and TTS for L ₂ )

● The user can correct / edit the automatically generated data as needed,

● Provide a variety of forms that allow the user to verify that the automatically generated translation is appropriate (ie, hear the pronunciation of the word through TTS).

If the user enters a word that does not match the predefined class in the system, the user can assign the word to the 'unknown' class. In ASR, this 'unknown' class is defined as words that occur in the learning data but not in the recognition vocabulary. In SMT, a bilingual entry that does not occur in the translated lexicon is set in an unknown tag in the target language model.

Improved probability and relevance within classes

Neither of these input methods requires language learning and does not provide an intuitive way for the user to determine whether a new word is properly represented. The user can then accept the new word entry by adding it to the user's personal lexicon, "Multilingual System Dictionary ". The entire system integrates standard vocabularies with custom vocabularies to create a user's runtime dictionary.

The in-class probability P (w | C) is also defined in addition to the five entries. In this way, the system can distinguish words belonging to the same class. So words that are closer to the user's work, preferences and habits will have priority and are assigned a higher probability in the class. This increase in probability in the higher class is determined based on the relevance to the user, which is evaluated according to observation.

New word entries and their recency

○ New words entered are likely to be used in the near future because they showed the user that they wanted the word by inputting the word, so the probability in the class is improved (increased) compared to the existing class entries.

● Correlate user activities, interests and tasks, including new words and:

○ Town names, landmarks, places of interest, etc.

History of past use

Coincident statistics (Sushi correlates better with Tokyo than Bogota)

● The general saliency of new words, including:

○ Urban population

○ Recent Opinion in Media

Such observations and relevance statistics are collected by observing the occurrence of new words in the system on a large background language resource, such as the Internet, and / or based on the user's observation location, history or activity. Such statistics can be collected in a data-rich language in a single language and applied to translation dictionaries and translation language models.

The user's new activities and tasks will be boosted if such information becomes less available over time and / or new information (such as arrival in another city) makes the subclass of the word less relevant the relevance of boosted words may decrease over time.

Cross-modal entry

Optionally, new words are entered according to one of the following:

● Speaking: The user speaks a new word. All information, such as pronunciation and translation, is assessed by a new word model, translation model, and background dictionary based on sound input, just as before. The system can participate in verbal conversations and select class identity and other relevant information.

● Spelling: The user spells the new word acoustically. This input method generally improves the accuracy of the translation more accurately than the word. This method may be supplemented by speech and other input forms.

● Handwriting: The user enters a new word by hand. This input method generally improves the accuracy of the translation more accurately than the word. This method may be supplemented by other forms of speech, spelling and other input.

● Browsing: New words may be selected according to interactive browsing. Where the system can present relevant new words relevant to the user by retrieving via the Internet a text having a statistical profile similar to the user's recent usage history and / or recently selected input new words.

Develop new remote word learning and shared vocabulary through internet

All of the methods described in the previous section are aimed at enabling individual users to tailor the speech translation system to their personal needs and tasks in the field. However, many of such customizations may also be useful to other users as well. In an embodiment, the customization may be uploaded to a community wide database in which names, jargon or expressions are shared among stakeholders. Vocabulary entries, translations, and class tags are collected and associated with communities with similar purposes. The following users can download these shared community resources and add them as resources to their systems.

Alternatively, users can choose to upload a roughly translated sentence and request a manual translation from the community. Other users may voluntarily (ie, free of charge) provide online proofreading and translation for such inaccurate or incomplete source words or phrases and their missing or inaccurate translations. These corrections and translations are submitted once again to the updated shared community translation database.

Unsupervised adaptation

After correction, correction and new word learning, the final corrected sample sentence and thus the original sentence or translation of the oral sentence are obtained. A voice translation device or system may use the fact that such ground truths have been provided to further adapt the ASR module (FIG. 1, module 2 or 9) to the main user of the device. Such adaptation is designed to improve the accuracy and usability of the device. Two specific adaptation methods are performed. The first is the adaptation of the system and the acoustic model and pronunciation model adaptation to better recognize the user's voice, and the second is adaptation to the user's speech through language model adaptation. Profiles are used to store adaptation data for a particular user and can be changed in the field.

Class-based machine translation

In the previous section, error correction and new word learning were described. These modules refer to class-based machine translation. The following describes the detailed functions of such class-based machine translation.

Approach

The current machine-level translation system performs translation at word-level. This is illustrated in the following three documents; (1) "P. Koehn, H. Hoang, A. Birch, C. Callison-Burch, M. Federico, N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin, and E. Herbst. 'Moses: Open source toolkit for statistical machine translation', In Proc. ACL, 2007 ("[Koehn07"); (2) "D. Chiang, A. Lopez, N. Madnani, C. Monz, P. Resnik and M. Subotin, "The Hiero machine translation system: extensions, evaluation, and analysis," In Proc. Human Language Technology and Emperical Methods in Natural Language Processing, pp. 779-786, 2005 ("Chiang05"); And " K. Yamada and K. Knight "A decoder for syntax-based statistical MT". In Proc. Association for Computational Linguistics, 2002 ("Yamada02"). Word sorting is performed; The translation sample or phrase pairs are matched at the word level; And a word-based language model. Hierarchical translation modules such as those in Chiang05 and syntax-based translation models such as those in Yamada02 expand on the basis of introducing intermediate structures. However, these approaches still require exact word matching. Since each word is treated as an independent entity, these models are not generalized to invisible words.

One example of a class-based machine translation is a class-based statistical machine translation where a sample sentence ^ e ^I ₁ = f ₁ , f ₂ , ..., f _J with the maximum likelihood given by a foreign language sentence f ^J ₁ = by searching for other languages _{^{e I 1 = e 1, e}} 2, ..., e _I is translated.

클래스는 개체명과 같은 시맨틱 클래스, 신택틱 클래스 또는 등가적 단어 또는 단어 구들로 이루어진 클래스일 수 있다. 일례로서 개체명 클래스가 시스템에 포함된 경우에 대해서 설명한다.

A class can be a semantic class, a syntactic class, equivalent to an object name, or a class consisting of equivalent words or word phrases. As an example, the case where the entity name class is included in the system will be described.

The models that provide the two most useful information applied during translation are the target language model P (e ^I ₁ ) and the translation model P (f ^J ₁ | e ^I ₁ ). Class-based statistical machine translation system, _{^{P (f J 1 | e I}} 1) is a class-based translation model (Fig. 3, model 23) and P (e ^I ₁₎ is a class-based language model (Fig. 3, model (24) )to be.

A class-based model for the statistical machine translation system can be learned using the procedure shown in FIG. First, the learning corpus of the sentence pair is normalized (step 100) and the corpus is tagged with the tagging model (FIG. 3, model 22) (step 101). One way to do this is described in Lafferty01. At this stage, sentences combined to form a learning pair can be tagged independently or collectively, and tags from one language can be projected into another language. After the tag is attached to the entire learning corpus, the words in the sentence pair are aligned (step 102). Sorting is "Franz Josef Och, Christoph Tillmann, Hermann Ney:" Improved Alignment Models for Stastical Machine Translation "; pp. 20-28; Proc. of the Joint Conf. of Emperative Methods in Natural Language Processing and Very Large Corpora; University of Maryland, College Park, MD, June 1999 "; And Brown Peter F., Stephen A. Della Pietra, Vincent J. Della Pietra, and R. L. Mercer. 1993. "The mathematics of statistical machine translation: Parameter estimation," Computatiional Linguistics, vol 19 (2): 263-311. At this stage, the multi-word phrases within the tagged entity (ie, "New York") are treated as a single token. Next, a phrase is extracted using a method such as Koehn07 to generate a class-based translation model (FIG. 3, model 23) (step 103). A tagged corpus is also used to learn the class-based object language model (FIG. 3, model 24). Learning is "B. Suhm and W. Waibel, "Towards better language models for spontaneous speech" in Proc. Can be accomplished using procedures such as those described in ICSLP-1994, 1994 ("Suhm94") (step 104).

The method shown in Fig. 11 is applied to translate the input sentence. First, the input sentence is normalized (step 105) and tagged (step 106) using a similar procedure as applied to the learning corpus. This input is tagged using a single language tagger (FIG. 3, model 22). Next, this input sentence is decoded using a class-based MT model (FIG. 3, models 23 and 24). For class-based statistical machine translation, decoding is performed using the same procedure used for standard statistical machine translation. However, the phrase pair matches the class level, not the word, as shown in the example below.

If the tagged input sentence is given as follows,

the train to @PACE.city {Wheeling} leaves at @TIME {4:30}

The following phrases can be matched.

A word or phrase in a class (ie, @ PLACE.city {Wheeling}, @ TIME {4:30}) is immediately passed (for number / time) or translation is determined from the translation model. The user can add new words to the translation model via the "user field customization module" (FIG. 1, module 12). If the user has already added the city name "Wheeling " (as detailed in the example of Fig. 6), the translation model will also include the following phrase.

If the MT model-based language model probability is given by P (e ^I ₁ ) (FIG. 3, model (24)) with the translation model probability P (f ^J ₁ | e ^I ₁ ) A search is performed to find a translation sample with probability P (f ^J ₁ | e ^I ₁ ) P (e ^I ₁ ).

Given the input sentences and phrases, the translation generated would be:

In this example, even though the word "Wheeling" does not appear in the learning corpus, once the user has entered the word through the " user field customization module "(FIG. 1, module 12) . Furthermore, since we know the word class ("@ PLACE.city" in this example), the system will be able to select a better translation for the surrounding words and arrange them correctly in the translation output,

Parallel Tagging of Multilingual Corpus

In an embodiment, labeled parallel corpus is obtained by independently tagging a single language tag to each side of the learning corpus and removing inconsistent labels in each sentence pair. In this method, the label sequence pair Ta, Tb having the maximum condition probability P (Ta, Sa) and P (Tb, Sb) is selected for each sentence pair Sa, Sb. If the number of occurrences of an arbitrary class tag is different between P (Ta, Sa) and P (Tb, Sb), the class tag is removed from the label sequence pair (Ta, Tb). One way to evaluate P (Ta, Sa) and P (Tb, Sb) is to apply a conditional random field-based tagging model (Lafferty01). An example of the features used during single language tagging is shown in FIG.

In an embodiment, labeling consistency for sentence pairs can be further improved using a target word extracted from word alignment (wb, j in FIG. 11) in addition to a single language feature.

In another embodiment, all of the sentences in the translation pair are labeled jointly by applying a restriction that the set of class tags should be the same. Specifically, the pair of labels Sa and Sb is searched for a pair of labels Ta and Tb that maximizes the joint maximum condition probability.

? aP (Ta, Sa)? bp (Tb, Sb) where Oi (Ta) = Oi (Tb)

Oi (Ta) Number of occurrences of class tag i in label sequence Ta

            (Actual number, not word count)

Total number of M classes

λa, λb Scale factor

If the performance of single language models is significantly different, λa and λb can be optimized to improve the tagging performance of the two languages.

A label may be generated by projecting a label from a first language that knows the label across the sentence pairs in the learning corpus to the annotation-free language if the manual annotation-attached corpus is not available for the particular language in the embodiment. One way to do this is "D. Yarowsky, G. Ngai and R. Wicentowski, "Inductive Multiligual Text Analysis TOOLS via Robust Projection across Alighed Corpora," In Proc. HLT, pages 161-168, 2001 ("Yarowsky01").

Evaluation of example systems and class-based machine translation

Experimental evaluation shows that class-based machine translation as described above improves translation performance over the conventional approach. Moreover, the accuracy of translation is improved by using the parallel tagging method described in Section 2.2.2.

We evaluated the Japanese translation system developed for tourism. Table 1 shows the description of the learning and test data.

English Japanese Parallel learning corpus Number of sentence pairs 400k Number of tokens 3,257k 3,171k Average sentence length 8.7 8.5 Manual tagged learning data (a subset of the above data) Learning (pair of sentences) 12600 Held-out test (no. Sentence pair) 1400 Test set Number of sentence pairs 600 Number of tokens 4393 4669 Average sentence length 7.3 7.8 OOV rate 0.3% 0.5%

Learning and testing data

Accurate and consistent tagging of sentence pairs is essential to implementing an effective class-based SMT. We investigated two ways to improve the quality of tagging. These methods are: first, introducing two language features from word alignment; and second, two-language tagging, jointly tagging both sides of a sentence pair. In the parallel learning corpus, 14,000 sentence pairs were manually tagged using the 16 class labels shown in Table 2.

class Class Labels number Radix, ordinal, sequence, character time Time, date, day, month Person First Name, Last Name Place City, Countryside, Landmark Agency Airlines, hotels, companies

Classes used in the evaluation system

Of these sets of manual labeling, 10% (1400 sentence pairs) were selected that included one or more tags as held-out data to evaluate tagging accuracy.

First, we evaluated the performance of a reference monolingual CRF-based tagger. Each side of the handout data set was labeled independently using a language dependent model. The output was then compared to the manual reference. Table 3 shows the tagging accuracy for various weighings.

Tagging method
English Japanese Two % Correctly tagged sentence pair
P R F P R F P R F Single language 0.95 0.89 0.92 0.94 0.88 0.91 0.88 0.80 0.84 80% + Sort feature 0.97 0.85 0.91 0.98 0.93 0.95 0.95 0.82 0.88 82% + Remove non-consistent tags 0.99 0.83 0.90 0.99 0.82 0.90 0.99 0.81 0.89 82% Two language tagging 0.98 0.92 0.95 0.98 0.92 0.95 0.97 0.90 0.93 92% + Sort feature 0.98 0.93 0.96 0.98 0.93 0.96 0.98 0.92 0.95 92%

Single-language and two-language tagging accuracy for hands-off learning sets

For two-language tagging, the tag is considered correct if the label is correctly attached to both sides of the corpus to the entity. The right column shows the percentage of sentence pairs to which tags are correctly attached on both sides. For independent languages, the F score is greater than 0.90, but the two-language tagging accuracy is much lower, at 0.84, and only 80% of the sentence pairs are correctly tagged. Integrating the alignment feature with a single language tagger improves accuracy for both languages and significantly improves recall for the Japanese side, but the proportion of correctly tagged sentence pairs has only slightly increased. Removing the non-coherent tag for the sentence pair improved the accuracy but did not increase the number of correctly tagged sentence pairs.

Next, the effects of the two-language tagging were evaluated using the above-described method. The tagging accuracy of this method is shown in the bottom two rows of Table 3 when the word order feature is incorporated. Compared with the case of a single language, the two-language tagging greatly improved the tagging accuracy. In addition to improving tagging consistency (F scores for two languages increased from 0.84 to 0.95), the tagging accuracy of both English and Japanese has also increased. Consolidating the word alignment features slightly improved tagging accuracy in all measurements.

We compared the performance of three class - based systems and the performance of the reference system without the class model to evaluate the effectiveness of the system.

Reference systems are described in detail in the Moses toolkit as described in Koehn 05 (" Franz Josef Och, Hermann Ney. A Systematic Comparison of Various Statistical Alignment Models ", Computational Linguistics, volume 29, number 1, pp. 19-51 March 2003 GIZA ++ was used to study the phrase - based translation model. The 3-gram language model is "A. Stolcke "SRILM - an extensible language modeling toolkit ", In Proc. of ICSLP, pp. 901-904, 2002 ". The decoding was performed using the PanDoRA decoder of the present application. This decoder is described in "Ying Zhang, Stephan Vogel," PanDoRA: A Large-scale Two-way Statistical Machine Translation System for Hand-Held Devices, "In the Proceedings of MT Summit XI, Copenhagen, 10-14 2007 " The systems were written for both translation directions J → E (Japanese to English) and E → J (Japanese to Japanese) using the learning set described in Table 1. The data used to train the target language model was limited to this corpus. The translation quality of the reference system was evaluated against a test set of 600 sentences. One reference was used during the evaluation. The BLEU scores for the J → E and E → J systems were 0.4381 and 0.3947, respectively. The BLEU score is "Kishore Papineni, Salim Roukos, Todd Ward and Wei-Jing Zhu", BLEU: a Method for Automatic Evaluation of Machine Translation, "In Proc. Association for Computational Linguistics, pp. 311-318, 2002 ". We evaluated the translation quality using the following three different tagging methods.

+ num: Number, time related to 8 classes

+ NE-class: More than +8 classes for object names

+ Bi-Tagging: learning corpus tagged with more than 16 classes in 2 languages

For + num and + NE-class cases, single-language tagging was applied and inconsistent tags were removed in sentence pairs. In case of + Bi-Tagging, we used two language tagging which integrated word alignment features. For each tagging scheme, the appropriate set of class labels was tagged for the entire learning corpus. The class-based translation and language model was then taught using the same procedure used in the reference system. During the test, the tag was attached to the input sentence using a single language tag. All object names in the test set were entered into the user dictionary to be used during translation.

The performance of the 600-sentence test set for the reference system and the class-based system is shown in Table 4 as a BLEU score for the J → E and E → J systems.

system
Translation quality (BLEU [Papineni02]) J → E E → J standard 0.4381 0.3947 + num 0.4441 0.4104 + NE-class 0.5014 0.4464 + Bi-Tagging 0.5083 0.4542

Translation quality of class-based SMT

A class - based SMT system (+ num) using number and time tags has improved translation quality compared to the reference system for both translation directions. For these models, the BLEU score was 0.4441 and 0.4104. In addition to the number and time tags, the translation quality is greatly improved when a class-based system using the object name class is applied. For the J → E system, the BLEU score was 0.5014 and for the E → J system, the BLEU score was 0.4464. When tagging the learning corpus using two language tagging (+ Bi-Tagging), BLEU increased by 0.8 points for both translation directions. For 14% of the sentences in the test set containing more than one entity name, the (+ Bi-Tagging) system performed better than the single-language tagging system ("NE-class") by 3.5 BLEU points.

While the present invention has been described in considerable detail, it should be understood that the drawings and detailed description have been presented for purposes of illustration only and are not intended to limit the invention. Various design and configuration changes are possible, but these are also within the principles of the present invention. Those skilled in the art will appreciate that such modifications and variations of the present invention, combinations of elements, alterations, equivalents, or improvements are also within the scope of the invention as defined in the appended claims.

Claims (45)

A method for updating a vocabulary of a speech translation system for translating a first language into a second language,
Receiving, by at least one microphone of the speech translation system, utterance from a user of the speech translation system, the speech translation system translating the speech from the first language into the second language, To output an audible translation of the speech in the second language from at least one speaker of the speech translation system;
After receiving said speech, from a user of said speech translation system, via a user interface of said speech translation system, in a first language lexicon of said first language of an automatic speech recognition module of said speech translation system, Receiving an indication to add a new word, the automatic speech recognition module for the speech translation system comprising: a first recognition lexicon; an acoustic model for the first language; and a language for the first language Model, wherein the new word is not included in the first recognized lexicon of the first language, the acoustic model, and the language model;
Determining, by the speech translation system, word class information for the new word, pronunciation in the first language, and translation in a second language;
The new word is translated by the speech translation system into the first recognition lexicon of the first language of the first language of the speech translation system with the word class information determined by the speech translation system and the pronunciation in the first language ;
And translating the new word into a first machine translation module associated with the first language of the speech translation system, with the word class information determined by the speech translation system and the translation into the second language, Wherein the first machine translation module includes a first tagging module, a first translation model, and a first language module, the translation module translating the new word into a corresponding translation word in the second language Configured -
Wherein the speech translation system comprises: The method according to claim 1,
Adding the new word to the first recognition lexicon of the first language and adding the new word to the first machine translation module are performed without reinitializing or restarting the automatic speech recognition module In a voice translation system. The method according to claim 1,
Wherein the step of adding the new word in the first machine translation module is performed without reinitializing or restarting the first machine translation module. The method according to claim 1,
Translating the corresponding translation word from the second language back to the new word of the first language by a second machine translation module associated with the second language of the speech translation system;
Correlating the new word in the first language with the corresponding translation word in the second language, and adding the corresponding translation word and the word class information to a second recognition lexicon of the second language; And
Updating the second machine translation module with the corresponding translation word and the word class information, the second machine translation module including a second tagging module, a second translation model and a second language module,
Further comprising the steps of: The method according to claim 1,
Inputting the corresponding translation word into a text-phonetic pronunciation lexicon associated with the second language
Further comprising the steps of: 5. The method of claim 4,
Inputting the new word into a text-to-speech pronunciation lexicon associated with the first language
Further comprising the steps of: 5. The method of claim 4,
Wherein the step of adding the corresponding translation word to the second recognition lexicon further comprises increasing the relative word probability of the new word in the class of the class based language model associated with the second language, A method of updating a lexicon of The method according to claim 1,
Translating the new word of the first language into a corresponding translation word of the second language and one or more other languages;
Correlating the new word with a corresponding third or more words of the one or more other languages, respectively;
Adding the third or more words of the one or more other languages to an aware vocabulary associated with each of the one or more other languages of the speech translation system; And
Updating machine translation modules associated with the one or more other languages, each machine translation module including a tagging module, a translation model and a language module,
Further comprising the steps of: The method according to claim 1,
Recognizing the new word in the speech received from the user using the reliability measurement and the new word model by the speech translation system; And
Prompting the user to add the new word to the user through the user interface of the voice translation system
Further comprising the steps of: The method according to claim 1,
Wherein adding the new word to the first recognition lexicon of the first language further comprises increasing the relative word probability of the new word in the class of the class based language model associated with the first language. A method for updating a lexical item in a translation system. 11. The method of claim 10,
Increasing the relative word probability of the new word associated with the first language is performed outside the known class by associating the new word with an unknown class and increasing the probability within the class of the unknown word Of the speech translation system. 11. The method of claim 10,
Wherein adding the new word to the first recognition lexicon of the first language further comprises increasing the translation probability of the new word. The method according to claim 1,
Wherein the step of determining the word class information comprises accepting word class information provided by the user. The method according to claim 1,
Wherein the step of determining the word class information comprises selecting one or more possible descriptions from the dictionary associated with the speech translation system and displaying the one or more possible descriptions for user acceptance. . The method according to claim 1,
Wherein the step of determining the word class information comprises automatically generating a hypothesis using a user field customization module of the speech translation system. 16. The method of claim 15,
Wherein the step of generating the sample statement is learned from the translated speech data. 16. The method of claim 15,
Further comprising selecting, by the speech translation system, a most probable word class for the new word based on co-occurrence statistics of other words having a similar known class How to update the vocabulary. The method according to claim 1,
After said step of determining said pronunciation and said translation for said new word, said user interface of said speech translation system inducing said user to said pronunciation of said new word and said translation; And
Upon receipt of said pronunciation for said new word and verification of said translation from said user:
Adding the new word to the first recognized lexicon of the first language of the speech translation system, along with the word class information determined by the speech translation system and the pronunciation in the first language; And
Adding said new word to said first machine translation module associated with said first language of said speech translation system with said word class information determined by said speech translation system and said pronunciation in said second language;
Wherein the speech translation system comprises: 19. The method of claim 18,
Wherein the speech translation system determines the word class information, pronunciation and translation for the new word without inducing the user input requiring expertise in computer speech and language processing techniques How to update the vocabulary. The method according to claim 1,
Wherein the speech translation system determines the word class information using the first tagging module. 21. The method of claim 20,
Wherein the speech translation system determines the word class information based on an ambient context for the new word. The method according to claim 1,
Wherein the step of determining the pronunciation of the new word in the first language comprises determining the pronunciation of the new word in the first language by machine learning. 23. The method of claim 22,
Wherein the step of determining the translation of the new word in the second language comprises translating the translation of the new word in the second language by statistical machine translation based on the pronunciation of the new word in the first language And determining a vocabulary of the speech translation system. 24. The method of claim 23,
Determining a pronunciation of the new word in the second language based on the translation of the new word in the second language by the speech translation system
Further comprising the steps of: In a field-maintainable class-based translation apparatus,
At least one microphone for receiving audible speech in a first language from a user of the device;
An automatic speech recognition module for the first language communicating with the at least one microphone, the automatic speech recognition module comprising a recognition vocabulary for the first language, an acoustic model for the first language, and a language model for the first language Automatic speech recognition module;
A machine translation module for communicating with the automatic speech recognition module for translating the output from the automatic speech recognition module in the first language into a second language;
A user interface for a user to input an indication to add a new word to the recognition lexicon of the automatic speech recognition module for the first language; And
A correction and calibration module for determining word class information for the new word, pronunciation in the first language, and translation in a second language,
/ RTI >
Wherein when the new word is not included in the recognition lexicon of the first language and is not included in the language model of the first language, the new word is updated by the word class information determined by the correction and correction module, With the pronunciation in one language being added to the recognition lexicon for the first language;
Wherein the new word is translated by the machine translation module into a corresponding translation word in the second language with the word class information determined by the correction and correction module and the translation in the second language, A field-maintainable class-based translation device that is added as a machine translation module. 26. The method of claim 25,
A second machine translation module associated with the second language for translating a second new word in the second language into a second translated word in the first language,
Further comprising: a field-maintainable class-based translation device. 26. The method of claim 25,
Wherein the user interface accepts an orthographic input in the language of the user. 26. The method of claim 25,
Wherein the language model of the first language is updated with at least one update based on a correction to an error made in the speech recognition sample sentence and the at least one update increases the language model probability of the corrected word sequence, and updating a probability in the language model for the first language of the automatic speech recognition module to reduce the likelihood of occurrence of the same error by reducing the probability of occurrence of a close-competing sample language model. Based translation device. 26. The method of claim 25,
An automatic speech recognition module for the second language for communicating with at least one microphone for recognizing speech in a second language received by the at least one microphone, An automatic speech recognition module including an acoustic model for two languages and a language model for the second language;
A machine translation module for communicating with the automatic speech recognition module for translating the output from the automatic speech recognition module in the second language into a first language; And
A second text-to-speech module for communicating with the machine translation module for the second language to generate an audible translation of the horses in the first language,
Further comprising: a field-maintainable class-based translation device. 26. The method of claim 25,
A text-to-speech module in communication with the machine translation module for generating an audible translation of the speech in the second language;
Further comprising: a field-maintainable class-based translation device. 31. The method of claim 30,
At least one speaker in communication with the text-to-speech module for outputting an audible translation of the speech in the second language
Further comprising: a field-maintainable class-based translation device. delete delete delete delete delete delete delete delete delete delete delete delete delete delete

Images