EP2274742A1 - Système et procédés pour maintenir une traduction orale-orale dans le domaine - Google Patents

Système et procédés pour maintenir une traduction orale-orale dans le domaine

Info

Publication number
EP2274742A1
EP2274742A1 EP09732921A EP09732921A EP2274742A1 EP 2274742 A1 EP2274742 A1 EP 2274742A1 EP 09732921 A EP09732921 A EP 09732921A EP 09732921 A EP09732921 A EP 09732921A EP 2274742 A1 EP2274742 A1 EP 2274742A1
Authority
EP
European Patent Office
Prior art keywords
language
word
translation
new word
new
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
EP09732921A
Other languages
German (de)
English (en)
Inventor
Ian R. Lane
Alexender Waibel
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Meta Platforms Inc
Original Assignee
Mobile Technologies LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Mobile Technologies LLC filed Critical Mobile Technologies LLC
Priority claimed from PCT/US2009/040677 external-priority patent/WO2009129315A1/fr
Publication of EP2274742A1 publication Critical patent/EP2274742A1/fr
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/08—Speech classification or search
    • G10L15/18—Speech classification or search using natural language modelling
    • G10L15/1815—Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/237—Lexical tools
    • G06F40/242—Dictionaries
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/40—Processing or translation of natural language
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/40—Processing or translation of natural language
    • G06F40/42—Data-driven translation
    • G06F40/44—Statistical methods, e.g. probability models
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/26—Speech to text systems
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L13/00—Speech synthesis; Text to speech systems
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/06—Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice
    • G10L15/063—Training
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/08—Speech classification or search
    • G10L15/18—Speech classification or search using natural language modelling
    • G10L15/183—Speech classification or search using natural language modelling using context dependencies, e.g. language models
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue

Definitions

  • the present invention is directed generally to speech-to-speech translation systems for cross-lingual communication, and more particularly, to a method and apparatus for field maintenance that enables users to add new vocabulary items and to improve and modify the content and usage of their system in the field, without requiring linguistic or technical knowledge or expertise.
  • ASR Automatic speech recognition
  • MT machine translation
  • the present invention solves the foregoing problems by providing a method and apparatus for updating the vocabulary of a speech translation system.
  • a method for updating the vocabulary of a speech translation system for translating a first language into a second language including written and spoken words.
  • the method includes adding a new word in the first language to a first recognition lexicon of the first language and associating a description with the new word, wherein the description contains pronunciation and word class information.
  • the new word and description are then updated in a first machine translation module associated with the first language.
  • the first machine translation module contains a first tagging module, a first translation model and a first language module, and is configured to translate the new word to a corresponding translated word in the second language.
  • the method additionally includes the steps of translating the translated word from the second language back into the new word of the first language, correlating the new word with a corresponding translated word of the second language and adding the translated word and its description to a second recognition lexicon of the second language.
  • a second machine translation module associated with the second language is then updated with the translated word and the description.
  • the second machine translation module contains a second tagging module, a second translation model and a second language module
  • the method further comprises the further step of inputting the first word into a text-to-speech pronunciation lexicon associated with the first language, and inputting the second word into a text-to-speech pronunciation lexicon associated with the second language.
  • the input signals may be of different modalities (eg. speech and nonverbal spelling, speech and verbal spelling, writing and speech, etc.) (referred to herein as "cross- modal") or may be of the same modality (speech and respeaking, writing and rewriting, etc.).
  • An embodiment of the invention is directed to a field maintainable class-based speech-to-speech translation system for communicating between a first language and a second language.
  • the system includes two speech recognition units, each configured for accepting sound comprising the spoken word of the first or second language and for producing text that corresponds to the spoken language, and two corresponding machine translation units, each configured to receive text from one of the speech recognition units and output a translation of the text into text of the other language. It also includes a user field customization module that enables the system to learn new words in cooperation with the user. The user field customization module is configured for accepting user selected input that comprises sounds or text corresponding to one or both of the languages and updates the machine translation units appropriately with the user selected input. [00014] In an embodiment, four primary features equip the system to provide a field maintainable class-based speech-to-speech translation system.
  • the first includes a speech translation framework that enables the addition of new words to the active system vocabulary, or the switching between location or task specific vocabularies. This provides for dynamic addition of words to a speech recognition module without requiring the module to be re-started.
  • the system uses multilingual system-dictionary and language independent word-classes across all system components in the speech-to-speech translation device, class- based machine-translation (phrase-based statistical MT, syntactic, example-based, etc,), multilingual word-class tagging during model training, based on combination of monolingual taggers, and word-class tagging in new language by way of alignment via parallel corpus from known tagged language.
  • a multimodal interactive interface enables nonexperts to add new words to the system.
  • the system is designed to accommodate ASR and SMT model adaptation using multimodal feedback provided by the user.
  • the system has networking capability to enable sharing of corrections or words.
  • a multimodal interactive interface enabling a user to add new words to a speech-to-speech translation device in the field and without technical expertise. Examples include: (1) Methods to automatically classify class of word or word-phrase to be added to the system, and automatically generate of pronunciations, and translation of the word; (2) Method for entering new words cross-modally by one or more of speaking, typing, spelling, handwriting, browsing, paraphrasing; (3) Multimodal feedback to help a linguistically untrained user determine if phonetic transliteration and translation is adequate: multiple textual forms (i.e.
  • an online system that corrects via multimodal user feedback in the field. Examples include: (1) Interface and methods to enable users to correct automatic speech recognition results, and use of this feedback information to adapt speech recognition components; (2) Interface and methods to enable users to correct machine translation hypotheses, and use of this feedback information to improve machine translation components; and (3) Method for automatically adjusting (enhancing or decreasing) language model, dictionary and translation model probability for correct or corrected word based on user correction.
  • an internet application that allows users to share corrections or new word additions done in the field across devices. Examples include: (1) Methods to upload, download and edit models for use in speech-to-speech translation devices via the world-wide-web; (2) Methods to collate in-the-field new words additions and corrections across the entire community of users; and (3) Methods to upload, download and edit, location or task specific vocabularies for use in speech-to-speech translation devices.
  • FIG. 1 is a block diagram illustrating a speech-to-speech translation system constructed according to an embodiment of the present invention
  • FIG. 2 illustrates an example of a graphical user interface that is displayed to user via a tablet interface
  • FIG. 3 is a flow chart illustrating the steps of speech-to-speech translation performed according to an embodiment of the present invention in FIG. 1 ;
  • FIG. 4 is a flow chart illustrating the steps by which the system learns from corrections made by the user (Correction and Repair Module);
  • FIG. 5 is a flow chart illustrating the steps by which users can add new words to system (User Field Customization Module);
  • FIG. 6 is a flow chart illustrating one example of methods by which the apparatus automatically generates the translation and pronunciations for new words the user wishes to add to the system;
  • FIG. 7 is a flow chart illustrating one example of a method to verify new word input via a multimodal interface
  • FIG. 8 illustrates an example of a visual interface to display automatically generated word information
  • FIG. 9 is a flow chart illustrating the steps required to train class-based MT models
  • FIG. 10 is a flow chart illustrating the steps of applying class-based MT to an input sentence.
  • FIG. 11 is a diagram illustrating possible features used during word-class tagging via statistical or machine learning approaches.
  • Various embodiments of the present invention describe methods and systems for speech-to-speech translation. Embodiments may be used to adapt to the user's voice and speaking style via model adaptation. In further embodiments, the user can correct recognition errors and the system can explicitly learn from errors that the user corrected, thereby making it less likely that these errors occur again in the future.
  • the present invention enables the user to customize the vocabulary to his or her individual needs and environment by either adding new words to the system, or selecting predefined dictionaries that are optimized for a specific location or task. When adding new words a multimodal interface allows the user to correct and verify automatically generated translations and pronunciations. This allows the user to add new words to the system when the user has no knowledge of the other language.
  • Figure 1 illustrates a block diagram overview of an example of a field maintainable speech-to-speech translation system according to the present invention.
  • the system operates between two languages L a and L b .
  • This is the typical implementation of a speech-to-speech dialog system involving speech-to-speech translation in both directions, from L 8 to L b and from L b to L 3 .
  • the bi-directionality of this configuration is not a prerequisite for the present disclosure.
  • a uni-directional system from L 3 to L b , or a multi-directional system involving several languages Li ... L n could equally benefit from the present invention.
  • the system has two ASR modules 2 and 9, that recognize speech for L 3 and L b , respectively, and produce text corresponding to L 3 and L b , respectively using acoustic model 18, ASR class-based language model 19 and a recognition lexicon model 20 (shown in Figure 3).
  • ASR modules 2 and 9 that recognize speech for L 3 and L b , respectively, and produce text corresponding to L 3 and L b , respectively using acoustic model 18, ASR class-based language model 19 and a recognition lexicon model 20 (shown in Figure 3).
  • Other types of ASR modules which may be used include speech recognizers developed by IBM Corporation, SRI, BBN or at Cambridge or Aachen.
  • the system also includes two machine translation modules 3 and 8, which translate text from L a to L b and from L b to L 3 , respectively (module 11).
  • the MT module used in this example was the "PanDoRA" system developed at Mobile Technologies, LLC.
  • Other MT modules could be used such as those developed by IBM Corporation, SRI, BBN or at Aachen university.
  • Two text-to-speech engines, 4 and 7 each corresponding to one of the machine translation modules 3 and 8, are configured to receive text produced from a corresponding ASR unit.
  • the output text is transferred to the respective MT module, 3 or 8, that translate text from L 3 to L b and from L b to L 3 , respectively.
  • the TTS module generates audio output to convert at least one text word in L 8 to speech via an output device 5, such as a loud speaker, and at least one text word in L b to speech via device 5 or another output device, such as a loud speaker 6, respectively.
  • a Cepstral TTS module was used. Any TTS modules which support Windows SAPI (speech application programming interface) conventions could also be employed.
  • a correction and repair module 11 allows the user to correct the system output via multiple modalities; including speech, gesture, writing, tactile, touch-sensitive and keyboard interfaces, and enables the system to learn from the user's corrections.
  • the correction and repair module may be of the type such as that disclosed in U.S. Patent No. 5,855,000.
  • a user field customization module 12 provides an interface for users to add new vocabulary to the system, and can also select an appropriate system vocabulary for their current situation. For example, triggered by a change in location, as determined by the GPS coordinates indicating the current location of the device, or an explicit selection of task or location by the user.
  • the user can access the user field customization module 12 and interact with the system via a graphical user interface displayed on the screen (or active touch screen) of the device 13, and a pointing device 14, including a mouse or pen.
  • a graphical user interface displayed on the screen (or active touch screen) of the device 13, and a pointing device 14, including a mouse or pen.
  • FIG. 2 An example of a graphical user interface is shown in FIG. 2.
  • the device 13 displays the text of audio input of a L a and corresponding text in window 15.
  • Machine translation of text L a in the second language L b is displayed in window 16.
  • the same microphone and loud-speaker can be used for both languages.
  • microphones 1 and 10 can be a single physical device
  • speakers 5 and 6 can be a single physical device.
  • FIG. 3 A flow chart illustrating the operation of an example of the method of the present invention is shown in FIG. 3.
  • the speech recognition system is activated by the user at step 15b.
  • a button can be selected on the graphical user interface ( Figure 2, item 15b) or on an external physical button (not shown).
  • the user's speech (item 25) is then recognized by one of the ASR modules in step 27; module 2, if the user is speaking L 3 , and module 9 if the user is speaking L b .
  • the ASR modules 2 and 9 apply three models: acoustic model 18, ASR class-based language model 19 and a recognition lexicon model 20. These models are language specific and each ASR module contains its own set of models.
  • the resulting text of the user's speech is displayed via the GUI on the device screen 13 at step 28.
  • Translation is then applied via MT module 3 or 8 based on the input language
  • MT modules 3 and 8 apply three main models: a tagging or parsing [Collins02] model to identify word classes (model 22), a class-based translation model (model 23), and a class-based language model (model 24).
  • the tagging model 22 may be any suitable type of tagging or parsing model such as the types described in J. Lafferty, A. McCallum, and F.
  • the automatically generated translation ( Figure 2, item 16) is translated back into the input language via MT module 3 or 8 and displayed with parentheses under the original input as illustrated for example in Figure 2, item 15a. If the confidence of both speech recognition and translation are high (step 31) as determined by the ASR model, 2 or 9, and the MT module, 3 or 8, spoken output (item 26) is generated via loud speakers 5 or 6, via TTS modules 4 or 7 (step 33). Otherwise, the system indicates that the translation may be wrong via the GUI, audio and/or tactical feedback.
  • the specific TTS module used in step 33 is selected based on the output language.
  • the user may intervene during the speech-to-speech translation process in any of steps from 27 to 33 or after process has completed. This invokes the Correction and Repair Module module 11 at (step 35).
  • the correction and repair module 11 records and logs any corrections the user may make, which can be later used to update ASR modules 2 and 9 and MT modules 3 and 8 as described in detail further below in this document.
  • step 36 If the correction contains a new vocabulary item (step 36), or if the user enters the field customization mode to explicitly add a new word to the system in step 15c, or if a new word is automatically detected in the input audio using confidence measures or new word models, such as the method described in Thomas Schaaf, "Detection of OOV words using generalized word models and a semantic class language model," in Proc. of Eurospeech, 2001 in step 15d; the User Field Customization Module 12 is invoked.
  • This module 12 provides a multimodal interface to enable users to add new words to the active system vocabulary. When a new word or phrase is added by a user the ASR, MT and TTS models (items 17, 21 and 33a) are updated as required.
  • a common set of classes for example person names, place names, and organization names
  • ASR A common set of classes
  • MT This provides a system-wide set of semantic slots that allows new words to be added to the system.
  • the names, special terms and expressions that occur within these classes are the words that are most variable depending on different users' deployments, locations, cultures, customs and tasks, and thus they are in greatest need of user-customization.
  • the specific classes used are dependent on the application domain of the system.
  • the classes may include semantic classes for named- entities; person, place and organization names; or task-specific noun phrases; for example: names of foods, illnesses or medicines; and another open class for words or phrases that don't fit into any of the predefined classes. Syntactic classes or word equivalence classes such as synonyms could also be used.
  • application domains include, but are not limited to, tourist, medical, peace keeping, and the like.
  • classes required in the tourist application domain include names of persons, cities, foods and the like.
  • for a medical professional application classes required include names of diseases, medications, anatomical names, and the like.
  • classes required for a peace-keeping application include names of weapons, vehicles, and the like.
  • Correction and repair module 11 enables a user to intervene in the speech-to- speech translation process at any time.
  • the user may either identify and log an error, or, if he/she wishes, correct an error in the speech recognition or translation output.
  • Such user intervention is of considerable value, as it provides immediate correction in the human- human communication process, and opportunities for the system to adjust to user needs and interests and to learn from mistakes.
  • a flow diagram illustrating this error feedback functionality is shown in Figure 4. If the user is dissatisfied with a translation of an utterance (i.e. an error occurs) the user can log the current input (step 40). The system will save audio of the current utterance as well as other information to a log file. This can be accessed and corrected by the user at a later time, or can be uploaded to a community database to allow expert users to identify and correct errors.
  • the user can also correct the speech recognition or machine translation output via a number of modalities.
  • the user can correct the entire utterance, by re-speaking it or entering the sentence via a keyboard or handwriting interface.
  • a user can highlight an erroneous segment in the output hypothesis via the touch-screen, mouse or cursor keys and correct only that phrase or word, using the keyboard, handwriting, speech, or explicitly spelling out the word letter-for-letter.
  • the user can also select an erroneous segment in the output hypothesis via the touch screen and correct it by selecting a competing hypothesis in an automatically generated drop-down list, or by reentering it by speech, or by any other complementary modality (e.g., handwriting, spelling, paraphrasing, etc.).
  • These methods and how to suitably combine complementary repair actions build on methods proposed by Waibel, et al., in US Patent No. 5,855,000 for multimodal speech recognition correction and repair. Here they are applied to the speech recognition and translation modules of interactive speech translation systems.
  • step 44 the system first determines if the correction contains a new word (step 44). This determination is made by checking for the word in the recognition lexicon model 20 associated with each language, L 3 and L b . If the word is not found the system prompts the user to add the new word to the active system vocabulary if desired ( Figure 5, step 50). Otherwise, the probabilities in the ASR models ( Figure 3, item 17) are updated to reduce the likelihood of the same error occurring again. This can be performed in a discriminative manner where probabilities of the corrected word sequence are increased, and those of close-competing hypotheses are reduced. [00046] A user can also correct the machine translation output if they have sufficient language expertise.
  • the machine translation output is corrected by the user (step 45) and the correction contains a new word, then the user is prompted with a dialog enabling them to add the new word to the active system vocabulary ( Figure 5, step 50). If the correction only contains words which are already in the active system vocabulary, then the machine translation models ( Figure 3, item 21) are updated. Specifically, an implementation can be used, where phrases are extracted from the corrected sentence pair and these are folded into translation models. The target language model used can be updated in a similar way to the ASR case.
  • User field customization module 12 enables the system to learn new words in cooperation with the user. Prior systems do not allow users to modify vocabularies in speech- to-speech translation systems. Unlike prior systems, user field customization model 12 enables the user to make incremental modifications in a running system that are relatively easy to perform for a non-expert, with minimal or no knowledge of computer speech and language processing technology or of linguistics. Model 12 offers such field customization by providing and accepting certain easy-to-understand feedback from the user, and based on this feedback deriving all the necessary parameters and system configurations autonomously. Field customization module 12 accomplishes this through: 1) an intuitive interface for user- customization, and 2) internal tools that automatically estimate all the internal parameters and settings needed for user customization, thereby relieving the user from this burden. [00048] For unidirectional translation, the system processes a minimum of four pieces of information about the word or phrase to add a new word or phrase to the active system vocabulary. These include:
  • the system also requires input of the pronunciation of the new word in L b .
  • the L b enables the TTS to generate audio output and the ASR modul for L b to recognize the new word in reverse.
  • FIG. 5 A flow chart illustrating the steps of operation of the user field customization model 12 is shown, for example, in Figure 5.
  • a new word is encountered by the system, based on a corrective intervention via the correction and repair model 11 in the previous section, it will prompt the user (Figure 5, step 50) to determine if this word should be "learned", i.e., added to the active system vocabulary. If so, a word learning mode is activated and the field customization module 12 begins to act.
  • Note that field customization or new-word learning need not only result from error correction dialogs.
  • the user may also specifically choose to enter a word learning mode from a pull-down menu, to add a new word or a list of new words a priori.
  • New word learning could also be triggered by external events that cause a sudden need for different words, such as specialty terms, names, locations, etc. In all such instances, however, the system must collect the above information.
  • the system After the user indicates that he/she wishes to add a new word to the system vocabulary (step 50), the system first looks up a large external dictionary, which is either contained locally on the device, or is a dictionary service that can be accessed via the Internet, or is a combination of both.
  • the external dictionary consists of entries of word translation pairs. Each entry contains pronunciation and word-class information which enables the new word to be easily added to the active system vocabulary. Each entry also contains a description of each word-pair in both languages.
  • step 51 the system displays a list of alternative translations of the word with a description of each (step 52). If the user selects one of the predefined translations from the dictionary (step 53) , then user can verify the pronunciation and other information provided by the dictionary (step 53a), and the edit it if necessary. The new word is then added to the active system vocabulary. [00051] To add a new word to the active system vocabulary, three steps are required
  • step 59 First the word and its translation are added to the ASR recognition lexicons of modules 2 and 9 (step 59). The word is added to this recognition lexicon 20 along with the pronunciation(s) given by the dictionary. As the user has just entered this word its probability of occurrence is set to be greater than competing members of the same class within the ASR class-based language model 19. This is to make words that were specifically added by the user more likely.
  • the word and its translation are added to the MT models ( Figure 3, item 21), enabling the system to translate the new-word in both translation directions. Finally, the word is registered with the TTS pronunciation model ( Figure 3, model 33a), which enables the system to pronounce the word correctly in both languages.
  • the system will automatically generate the information required to register the word into the active system vocabulary, and will verify this information with the user.
  • the class of the new word is estimated via a tagging model ( Figure 3, model 22) using the surrounding word context if it is available (step 54).
  • the pronunciation and translation of the new word are automatically generated via either rule-based or statistical models (step 55).
  • the resulting information is then shown to the user via a multimodal interface (step 58).
  • the system prompts the user to verify (step 58) or correct (step 57) the automatically generated translation or pronunciation.
  • the new word is added to the active system vocabulary (steps 59, 59a, 59b).
  • the recognition lexicon 20 (which is typically stored as a tree-structure, within ASR Modules 2 or 9) is searched and then updated to include the new word. This enables the new word to be added to the recognition vocabulary dynamically, and it can thus be recognized, immediately, if spoken in the following utterance.
  • the ASR system does not need to be reinitialized or re-started as in prior systems.
  • a new word (specifically, "word + translation + word class") can be appended to the MT translation model (59a), the translation model 23 (which is can be stored as a hash-map within MT modules 3 and/or 8) is searched and an new translation-pair containing the new word its translation, and word class is appended.
  • word + translation + word class a new word
  • the translation model 23 which is can be stored as a hash-map within MT modules 3 and/or 8
  • word class is appended.
  • the translation of the word and pronunciations for both the word and its translation are required. Generating this information can be implemented as a three-step process as shown, for example, in Figure 6.
  • the pronunciation of the word is generated (step 60). Based on the character sequence of the word and its pronunciation, a translation is generated (step 61).
  • the pronunciation of the new word in the target language is generated (step 62) using information generated in previous steps. Two examples for generating this information using different techniques within a Japanese-English Field Maintainable S2S Translation System are shown on the left hand side of Figure 6.
  • Machine learning may be conducted by any suitable technique such as those described by Damper, R. I. (Ed.), Data-Driven Techniques in Speech Synthesis. Dordrecht, The Netherlands: Kluwer Academic Publishers (2001).
  • the transliteration of this word in Japanese is automatically generated via statistical machine transliteration (step 66), and the Japanese pronunciation is then generated via manually defined rules (step 67).
  • Transliteration may be accomplished by using any suitable statistical machine transliteration engine. Examples include those discussed by K. Knight and J. Graehl , Machine transliteration. Computational Linguistics 24 4 (1998), pp.
  • the user can verify the generated translation and pronunciation via audible output.
  • Alternatively written form may be used if considered more suitable for the user, given their native language (i.e. in "Hanyu Pinyin” for Chinese, or "Romaji” for Japanese if the user is an English speaker).
  • the user may edit the translation and/or pronunciation if required. Once approved by the user, the word and word characteristics are added to the multilingual system dictionary.
  • the system also eliminates the need for a translation of each new word that is added to the dictionary by automatically generating the required information with the assistance of interactive user input.
  • An example of a user interface is shown in Figure 3.
  • the system consults the user to confirm and verify the estimated linguistic information. This is done in an intuitive manner, so as not to presume any special linguistic or technical knowledge. Thus, a suitable interface is used. In the following we describe the user interaction during new word learning.
  • the user may select a "new-word" mode from the menu, or the new word learning mode could be invoked after a user correction has yielded a new/unknown word.
  • the window pane that appears he/she can now type the desired new word, name, special term, concept, expression.
  • the orthographic input in the user's language this can be character sets different from English, e.g., Chinese, Japanese, Russian, etc.
  • the system then generates a transliteration in Roman alphabet and the words predicted pronunciation. This is done by conversion rules that are either hand written or extracted from preexisting phonetic dictionaries or learned from transliterated speech data.
  • the user views the automatic conversion and can play the sound of the generated pronunciation via
  • TTS TTS.
  • the user may iterate and modify either of these representations (script, Romanized transliteration, phonetic transcription, and its sound in either language) and the other corresponding entries will be regenerated similarly (thus a modified transcription in one language may modify the transcription in the other).
  • the system further automatically selects the most likely word class that the new word belongs to based on co-occurrence statistics of other words (with known class) in similar sentence contexts.
  • the new word window pane also allows for a manual selection
  • the user can assign it to the 'unknown' class.
  • the 'unknown' class is defined by words that occurred in the training data but not in the recognition lexicon.
  • Neither of these input methods requires linguistic training and provides an intuitive way for the user to judge if a new word was suitably represented.
  • the user may then accept this new word entry by adding the word to a "multilingual system-dictionary", that is a user's individual lexicon.
  • the overall system merges standardized lexica with customized lexica into the user's runtime dictionary.
  • C) is also defined. In this fashion it is possible for the system to differentiate between words belonging to the same class. Thus words that are closer to the user's tasks, preferences and habits will be preferred and a higher intra-class probability assigned. This boosting of higher intra-class probability is determined based on relevance to the user, where relevance is assessed by observing:
  • Such observations and relevance statistics are collected based on the user's observed location, history or activity, and/or alternatively by observing the occurrence of the system's new word in a large background language resource such as the internet.
  • Such statistics may be collected mono-lingually, in a data-rich language and applied to the translation dictionary and translation language model.
  • the relevance of boosted words may also decay over time, as the user's new activities and tasks render such words less likely over time and/or if new information (as the arrival at a different city) make a subclass of words less relevant.
  • a new word is entered by one of the following:
  • Spelling User spells new word acoustically. This input method generally improves the likelihood of a correct transliteration over speaking it. It may also be used complementarily to speaking and other input modalities,
  • Handwriting User enters new word by handwriting. This input method generally improves the likelihood of a correct transliteration over speaking it. It may also be used complementarily to speaking, spelling, or other input modalities, • Browsing: New words may also be selected by interactive browsing. Here the system may propose related, relevant new words by searching the internet for texts with similar statistical profiles as the user's recent usage history and/or recent selected entered new words.
  • users may choose to only upload poorly translated sentences, to request manual translation from the community.
  • users can provide online correction and translation on a volunteer (or paid fee) basis.
  • the resulting corrections and translations are once again resubmitted into the updated shared community translation database.
  • the speech-to- speech translation device or system automatically can use the fact that such ground truth has been provided to further adapt the ASR modules ( Figure 1 , module 2 or 9) to the primary user of the device. Such adaptation is designed to improve the accuracy and usability of the device.
  • Two specific methods of adaptation are performed. First, adaptation of the system to better recognize the user's voice; acoustic model and pronunciation model adaptation, and second, adapting to the user's style of speech by way of language model adaptation. Profiles are used to store adaptation data for specific users and can be switched in the field.
  • Classes can be semantic classes, such as named-entities, syntactic classes or classes consisting of equivalent words or word phrases. As an example we describe the case when named-entity classes are incorporated into the system.
  • the two most informative models applied during translation are the target language model and the translation model
  • the translation model In a class-based statistical machine translation framework is a class-based translation model (FIG. 3, model 23), and P(e'i) is a class-based language model (FIG. 3, model 24).
  • Class-based models for a statistical machine translation framework can be trained using the procedure shown in FIG. 10.
  • the training corpora of sentence pairs are normalized (step 100) and tagging models (FIG. 3, model 22) are used to tag the corpora (step 101).
  • tagging models FIG. 3, model 22
  • LaffertyOl One approach to do this is described in LaffertyOl .
  • sentences that combine to form a training-pair can be tagged independently, tagged jointly, or tags from one language can be projected to the other.
  • words within sentence-pairs are aligned (step 102).
  • Alignment can be accomplished using current approaches such as those described by Franz Josef Och, Christoph Tillmann, Hermann Ney: "Improved Alignment Models for Statistical Machine Translation”; pp. 20-28; Proc. of the Joint Conf. of Empirical Methods in Natural Language Processing and Very Large Corpora; University of Maryland, College Park, MD, June 1999; and Brown, Peter R, Stephen A. Delia Pietra, Vincent J. Delia Pietra, and R. L. Mercer. 1993. "The mathematics of statistical machine translation: Parameter estimation,” Computational Linguistics, vol 19(2):263— 311. In this step, multi-word phrases within a tagged entity (i.e. "New York”) are treated as a single token.
  • phrases are extracted (step 103) using methods such as Koehn07 to generate class-based translation models (FIG. 3, model 23).
  • the tagged corpus is also used to train class-based target language models (FIG. 3, model 24). Training may be accomplished using a procedure such as that described in B. Suhm and W. Waibel, "Towards better language models' for spontaneous speech' in Proc. ICSLP-1994, 1994 (“Suhm94"). (step 104).
  • the input sentence is normalized (step 105) and tagged (step 106) using a similar procedure as that applied to the training corpora.
  • the input sentence is tagged using a monolingual tagger (FIG. 3, model 22).
  • the input sentence is decoded using class- based MT models (FIG. 3, models 23 and 24).
  • class-based statistical machine translation decoding is performed using the same procedure used in standard statistical machine translation, However, phrase-pairs are matched at the class-level, rather than the word, as shown in the example below.
  • @TIME # @TIME at @TIME # @TIME D the train to @PLACE.city # @PLACE.city O D O D D leaves at @TIME # D @TIME 0 D D D O O word or phrases within a class (i.e.: @PLACE.city ⁇ Wheeiing ⁇ , @ ⁇ iME ⁇ 4:30 ⁇ ) are either passed directly through, which is the case for numbers/times, or the translation is determined from the translation model. Users can add new words to the translation model via the "User Field Customization Module" (FIG. 1, module 12). If the user had previously added the city name "Wheeling" (as detailed in the example in FIG. 6), then the translation model will also contain the following phrase:
  • a labeled parallel corpora is obtained by independently tagging each side of the training corpora with monolingual taggers and then removing inconsistent labels from each sentence-pair.
  • the label-sequence-pair (Ta 5 Tb) is selected which has maximum conditional probabilities P(Ta 5 Sa) and P(Tb 5 Sb). If the occurrence count of any class-tag differs between P(Ta 5 Sa) and P(Tb 3 Sb) 5 that class-tag is removed from the label-sequence-pair (Ta 5 Tb).
  • One method to estimate P(Ta 5 Sa) and P(Tb,Sb) is by applying conditional random field-based tagging models LaffertyOl . An example of a feature set used during monolingual tagging is shown in FIG. 11.
  • labeling consistency across sentence-pairs can be further improved by using the target word extracted from word-alignment (wb,j in FIG. 11), in addition to monolingual features.
  • both sentences in the translation-pair are jointly labeled while applying the constraint that the class-tag sets must be equivalent.
  • ⁇ a and ⁇ b can be optimized to improve bilingual tagging performance.
  • labels can be generated by projecting labels from a first language where labels are known, across the sentence-pairs in the training corpora to the non- annotated language.
  • One approach to do this is described in D. Yarowsky, G. Ngai and R. Wicentowski, "Inducting Multilingual Text Analysis Tools via Robust Projection across Aligned Corpora, " In Proc. HLT, pages 161-168, 2001 ("Yarowsky 01").
  • Example System and Evaluation of Class-based Machine Translation [00087] Through experimental evaluation, we show that class-based machine translation, as detailed above, improves translation performance compared to previous approaches. Furthermore, we show that by using the parallel tagging approach described in section 2.2.2, translation accuracy is further improved.
  • Table 1 Training and Test Data [00089] To realize effective class-based SMT, accurate and consistent tagging across sentence-pairs is vital. We investigated two approaches to improve tagging quality; first, the introduction of bilingual features from word-alignment; and second, bilingual tagging, where both sides of a sentences-pair are jointly tagged. From the parallel training corpora 14,000 sentence-pairs were manually tagged using the 16 class labels indicated in Table 2.
  • a tag is considered correct if the entity is correctly labeled on both sides of the corpora.
  • the right hand column indicates the percentage of sentence-pairs in which both sides were tagged correctly.
  • the F-score is above 0.90 for the independent languages
  • the bilingual tagging accuracy is significantly lower at 0.84, and only 80% of the sentence-pairs were correctly tagged.
  • Incorporating alignment features into the monolingual taggers improved precision for both languages and significantly improvement recall for the Japanese side, however, the percentage of correctly tagged sentence-pairs increased only slightly. Removing inconsistent tags across sentence-pairs improved precision, but the number of correctly tagged sentence-pairs did not improve.
  • the decoder is described in Ying Zhang, Stephan Vogel, "PanDoRA: A Large-scale Two-way Statistical Machine Translation System for Hand-held Devices," In the Proceedings of MT Summit XI, Copenhagen, Denmark, Sep. 10-14 2007.
  • Systems were created for both translation directions J ⁇ E (Japanese to English) and E ⁇ J (English to Japanese) using the training set described in Table 1.
  • the data used to train the target language models were limited to this corpora.
  • the translation quality of the baseline system was evaluated on a test-set of 600 sentences. One reference was used during evaluation.
  • the BLEU-score for the J ⁇ E and E ⁇ J systems were 0.4381 and 0.3947, respectively.
  • BLEU- score is described in Kishore Papineni, Salim Roukos, Todd Ward and Wei-Jing Zhu "BLEU: a Method for Automatic Evaluation of Machine Translation, " In Proc. Association for Computational Linguistics, pp. 311-318, 2002. Translation quality using three different tagging schemes was evaluated:

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Computational Linguistics (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • Human Computer Interaction (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Probability & Statistics with Applications (AREA)
  • Machine Translation (AREA)

Abstract

L'invention concerne un procédé et un appareil pour mettre à jour le vocabulaire d'un système de traduction de discours pour traduire une première langue en une seconde langue comprenant des mots écrits et parlés. Le procédé consiste à ajouter un nouveau mot dans la première langue à un premier lexique de reconnaissance de la première langue et à associer une description du nouveau mot, la description contenant la prononciation et des informations concernant la classe dudit mot. Le nouveau mot et la description sont ensuite mis à jour dans un premier module de traduction par machine (MT) associé à la première langue. Le premier module de traduction par machine, qui contient un premier module de marquage, un premier modèle de traduction et un premier module de langue, est conçu pour traduire le nouveau mot en un mot traduit correspondant de la seconde langue. L'invention peut être éventuellement utilisée pour une traduction bidirectionnelle ou multidirectionnelle.
EP09732921A 2008-04-15 2009-04-15 Système et procédés pour maintenir une traduction orale-orale dans le domaine Ceased EP2274742A1 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US4507908P 2008-04-15 2008-04-15
US9258108P 2008-08-28 2008-08-28
US9381808P 2008-09-03 2008-09-03
PCT/US2009/040677 WO2009129315A1 (fr) 2008-04-15 2009-04-15 Système et procédés pour maintenir une traduction orale-orale dans le domaine

Publications (1)

Publication Number Publication Date
EP2274742A1 true EP2274742A1 (fr) 2011-01-19

Family

ID=43333124

Family Applications (1)

Application Number Title Priority Date Filing Date
EP09732921A Ceased EP2274742A1 (fr) 2008-04-15 2009-04-15 Système et procédés pour maintenir une traduction orale-orale dans le domaine

Country Status (5)

Country Link
EP (1) EP2274742A1 (fr)
JP (2) JP2011524991A (fr)
KR (1) KR101445904B1 (fr)
CN (1) CN102084417B (fr)
BR (1) BRPI0910706A2 (fr)

Families Citing this family (30)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8725496B2 (en) * 2011-07-26 2014-05-13 International Business Machines Corporation Customization of a natural language processing engine
KR101870729B1 (ko) * 2011-09-01 2018-07-20 삼성전자주식회사 휴대용 단말기의 번역 트리구조를 이용한 번역장치 및 방법
US8930189B2 (en) 2011-10-28 2015-01-06 Microsoft Corporation Distributed user input to text generated by a speech to text transcription service
US9257115B2 (en) 2012-03-08 2016-02-09 Facebook, Inc. Device for extracting information from a dialog
US9672209B2 (en) * 2012-06-21 2017-06-06 International Business Machines Corporation Dynamic translation substitution
CN103474062A (zh) * 2012-08-06 2013-12-25 苏州沃通信息科技有限公司 一种语音识别方法
US20140067366A1 (en) * 2012-08-30 2014-03-06 Google Inc. Techniques for selecting languages for automatic speech recognition
KR101364774B1 (ko) * 2012-12-07 2014-02-20 포항공과대학교 산학협력단 음성 인식의 오류 수정 방법 및 장치
NZ714571A (en) * 2013-06-03 2016-10-28 Mach Zone Inc Systems and methods for multi-user multi-lingual communications
JP6366166B2 (ja) * 2014-01-27 2018-08-01 日本放送協会 音声認識装置、及びプログラム
US9514376B2 (en) * 2014-04-29 2016-12-06 Google Inc. Techniques for distributed optical character recognition and distributed machine language translation
JP6262122B2 (ja) * 2014-11-18 2018-01-17 日本電信電話株式会社 翻訳学習装置、固有表現学習装置、方法、及びプログラム
CN113140215A (zh) 2015-01-16 2021-07-20 三星电子株式会社 用于执行话音识别的方法和设备
CN105244027B (zh) * 2015-08-31 2019-10-15 百度在线网络技术（北京）有限公司 生成谐音文本的方法和系统
CN107102990A (zh) * 2016-02-19 2017-08-29 株式会社东芝 对语音进行翻译的方法和装置
CN113407743A (zh) 2016-04-08 2021-09-17 北京三星通信技术研究有限公司 物体信息翻译、以及衍生信息获取方法和装置
CN106328146A (zh) * 2016-08-22 2017-01-11 广东小天才科技有限公司 一种视频的字幕生成方法及装置
CN107798386B (zh) * 2016-09-01 2022-02-15 微软技术许可有限责任公司 基于未标注数据的多过程协同训练
CN107818086B (zh) * 2016-09-13 2021-08-10 株式会社东芝 机器翻译方法和装置
CN107844470B (zh) * 2016-09-18 2021-04-30 腾讯科技（深圳）有限公司 一种语音数据处理方法及其设备
CN107170453B (zh) * 2017-05-18 2020-11-03 百度在线网络技术（北京）有限公司 基于人工智能的跨语种语音转录方法、设备及可读介质
CN107391500A (zh) * 2017-08-21 2017-11-24 阿里巴巴集团控股有限公司 文本翻译方法、装置及设备
CN107871500B (zh) * 2017-11-16 2021-07-20 百度在线网络技术（北京）有限公司 一种播放多媒体的方法和装置
KR102582360B1 (ko) * 2018-02-08 2023-09-25 주식회사 케이티 사용자 맞춤형 언어 모델을 생성하는 장치, 단말 및 방법
US20210294986A1 (en) * 2018-07-31 2021-09-23 Optim Corporation Computer system, screen sharing method, and program
KR102437008B1 (ko) * 2018-08-01 2022-08-26 김민철 번역 서비스 제공 장치 및 방법
CN111508484B (zh) * 2019-01-31 2024-04-19 阿里巴巴集团控股有限公司 语音数据的处理方法及装置
KR20200107057A (ko) 2019-03-06 2020-09-16 삼성전자주식회사 음성 인식 모델에서 사용되는 언어를 확장시키는 방법 및 음성 인식 모델을 포함하는 전자 장치
CN111858830B (zh) * 2020-03-27 2023-11-14 北京梦天门科技股份有限公司 基于自然语言处理的卫生监督执法数据检索系统及其方法
CN112420024B (zh) * 2020-10-23 2022-09-09 四川大学 一种全端到端的中英文混合空管语音识别方法及装置

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH03282776A (ja) * 1990-03-30 1991-12-12 Nec Corp 文書処理方式
JPH04319769A (ja) * 1991-04-18 1992-11-10 Toshiba Corp 通訳システム
JPH05197744A (ja) * 1992-01-17 1993-08-06 Nippon Hoso Kyokai <Nhk> 外国語ワードプロセッサ
JPH07129594A (ja) * 1993-10-29 1995-05-19 Toshiba Corp 自動通訳システム
JPH08212216A (ja) * 1995-02-08 1996-08-20 Toshiba Corp 自然言語処理装置および自然言語処理方法
JPH08329088A (ja) * 1995-05-31 1996-12-13 Sony Corp 音声入力翻訳装置
JP4017329B2 (ja) * 2000-09-14 2007-12-05 沖電気工業株式会社 機械翻訳システム
JP4135316B2 (ja) * 2000-12-15 2008-08-20 ブラザー工業株式会社 機械翻訳装置
JP2004094721A (ja) * 2002-09-02 2004-03-25 Matsushita Electric Ind Co Ltd 携帯電話装置
JP2004310256A (ja) * 2003-04-03 2004-11-04 Ricoh Co Ltd 翻訳辞書管理方法及びシステム、翻訳システム及びそのプログラム並びにそのプログラムを記憶した媒体
JP4431759B2 (ja) * 2005-01-18 2010-03-17 株式会社国際電気通信基礎技術研究所 未登録語自動抽出装置及びプログラム、並びに未登録語自動登録装置及びプログラム
JP2006318202A (ja) * 2005-05-12 2006-11-24 Suri Kaiseki Kenkyusho:Kk 翻訳装置、翻訳プログラム及び翻訳システム
JPWO2007097390A1 (ja) * 2006-02-23 2009-07-16 日本電気株式会社 音声認識システム、音声認識結果出力方法、及び音声認識結果出力プログラム

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See references of WO2009129315A1 *

Also Published As

Publication number Publication date
KR101445904B1 (ko) 2014-09-29
CN102084417A (zh) 2011-06-01
JP6345638B2 (ja) 2018-06-20
BRPI0910706A2 (pt) 2017-08-01
JP2011524991A (ja) 2011-09-08
CN102084417B (zh) 2014-05-21
JP2016053726A (ja) 2016-04-14
KR20110031274A (ko) 2011-03-25

Similar Documents

Publication Publication Date Title
US11972227B2 (en) Lexicon development via shared translation database
US8204739B2 (en) System and methods for maintaining speech-to-speech translation in the field
JP6345638B2 (ja) 現場にて音声−音声翻訳をメンテナンスするシステム及び方法
US9753918B2 (en) Lexicon development via shared translation database
EP2849178A2 (fr) Système et procédé de traduction vocale améliorée
US10073843B1 (en) Method and apparatus for cross-lingual communication
JP5819924B2 (ja) アジア文字を生成するための認識アーキテクチャ
US8924210B2 (en) Text processing using natural language understanding
US9449599B2 (en) Systems and methods for adaptive proper name entity recognition and understanding
Bassil et al. Post-editing error correction algorithm for speech recognition using bing spelling suggestion
Anastasopoulos Computational tools for endangered language documentation
EP3005152B1 (fr) Systèmes et procédés de reconnaissance et compréhension d&#39;entités de noms propres adaptatives
Dessloch et al. KIT lecture translator: Multilingual speech translation with one-shot learning
Prasad et al. BBN TransTalk: Robust multilingual two-way speech-to-speech translation for mobile platforms
WO2009151868A2 (fr) Système et procédés de gestion d’une traduction parole-parole sur site
US11900072B1 (en) Quick lookup for speech translation
Narayanan et al. Speech recognition engineering issues in speech to speech translation system design for low resource languages and domains
Yu Efficient error correction for speech systems using constrained re-recognition
Gregory Efficient Error Correction for Speech Recognition Systems using Constrained Re-recognition
Toselli et al. Multi-Modality for Interactive Machine Translation

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20101108

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK TR

AX Request for extension of the european patent

Extension state: AL BA RS

17Q First examination report despatched

Effective date: 20110325

DAX Request for extension of the european patent (deleted)
RIN1 Information on inventor provided before grant (corrected)

Inventor name: WAIBEL, ALEXENDER

Inventor name: LANE, IAN, R.

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: FACEBOOK, INC.

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: FACEBOOK, INC.

REG Reference to a national code

Ref country code: DE

Ref legal event code: R003

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN REFUSED

18R Application refused

Effective date: 20160119