WO2016013685A1 - Procédé et système permettant de reconnaître un discours comprenant une séquence de mots - Google Patents

Procédé et système permettant de reconnaître un discours comprenant une séquence de mots Download PDF

Info

Publication number
WO2016013685A1
WO2016013685A1 PCT/JP2015/071426 JP2015071426W WO2016013685A1 WO 2016013685 A1 WO2016013685 A1 WO 2016013685A1 JP 2015071426 W JP2015071426 W JP 2015071426W WO 2016013685 A1 WO2016013685 A1 WO 2016013685A1
Authority
WO
WIPO (PCT)
Prior art keywords
constraint
gesture
speech
word
sequence
Prior art date
Application number
PCT/JP2015/071426
Other languages
English (en)
Inventor
Bret Harsham
John Hershey
Original Assignee
Mitsubishi Electric Corporation
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US14/337,472 external-priority patent/US9196246B2/en
Application filed by Mitsubishi Electric Corporation filed Critical Mitsubishi Electric Corporation
Priority to DE112015003357.2T priority Critical patent/DE112015003357B4/de
Priority to JP2016554899A priority patent/JP6272496B2/ja
Priority to CN201580040168.8A priority patent/CN106537489B/zh
Publication of WO2016013685A1 publication Critical patent/WO2016013685A1/fr

Links

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/08—Speech classification or search
    • G10L15/18—Speech classification or search using natural language modelling
    • G10L15/183—Speech classification or search using natural language modelling using context dependencies, e.g. language models
    • G10L15/19—Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G06F3/04842—Selection of displayed objects or displayed text elements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition
    • G06V30/19—Recognition using electronic means
    • G06V30/196—Recognition using electronic means using sequential comparisons of the image signals with a plurality of references
    • G06V30/1983—Syntactic or structural pattern recognition, e.g. symbolic string recognition
    • G06V30/1985—Syntactic analysis, e.g. using a grammatical approach
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/24—Speech recognition using non-acoustical features
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/26—Speech to text systems

Definitions

  • the present invention relates generally to automatic speech recognition, and more particularly to providing the speech recognition while placing low cognitive load on a user.
  • Some methods for correcting unrestricted speech utilize editing applications, i.e., word processors with computer displays and conventional keyboards.
  • word processors with computer displays and conventional keyboards.
  • the method described in U.S. 5,960,447, corrects misrecognitions of speech by tagging words in the recognized text with associated speech and allowing users of word processing software to listen to the speech for editing text transcriptions.
  • Some embodiments of the invention are based on a realization that errors that occur in speech recognition of large-vocabulary continuous speech including a sequence of words are often associated with other errors. Therefore, a correction to the recognition result can be used to predict where other errors might have occurred and to correct the additional errors, instead of just correcting the initial error and waiting for additional user input. These other errors may occur in adjacent word locations, or they may occur in more distant locations without any changes needed to adjacent words, thus the correction should be used to update the recognition result.
  • the correction constraint can men be used to re-interpret the speech uttered by a user.
  • constraints can improve the probability of correctly interpreting group of words, as contrasted with an interpretation that only uses the acoustic signals corresponding to the individual words. For example, given a recognition result including alternates "Wreck a nice beach” and “Recognize speech", a constraint that the topic is "pollution” can be used to update the probabilities of the first recognition.
  • an input from the user can be used to determine the word sequence constraints, which can be used to update the entire
  • the word-sequence constraints include metadata of the speech, such as a number of words in the speech or the absence of a specific word in the speech.
  • the word-sequence constraints are more specific to the context of the speech.
  • the word- sequence constraints can include information about initial interpretation of the speech, and such information is used to re-evaluate the interpretation of the entire speech.
  • Some embodiments are also based on recognition that in order for the system to impose a low cognitive load on a user, the method(s) of entry of word sequence constraints should minimize the actions of the user required for selection of the constraints.
  • the word-sequence constraints can have different values, but also can have different types.
  • the types of the word-sequence constraints can include selection or rejection of a word, selection of the sequence of word, selection of a timeslot, a trace of correct words in the sequence, replacement of words or order of words, etc. Accordingly, it is desired to provide an interface adapted for selection of both type and value of the constraints with minimal destruction of the users from their primary task, such as driving.
  • Some embodiments of the invention are based on a realization that a single gesture of the user can be analyzed to determine both the type and the value of the word-sequence constraints.
  • a shape of the gesture can be used to determine a type of the word-sequence constraint, and a location of the gesture with respect to the interface, e.g., a location of the gesture path as mapped to the coordinate space of the user interface (and possibly as displayed on an output device), can be used to determine the value of the word-sequence constraint.
  • a horizontal gesture can be mapped to the type of the word- sequence constraint of selecting a sequence of words.
  • the location of the path of this gesture on the display device can be used to determine the sequence of words present in the interpretation of the speech.
  • a vertical gesture can be mapped to the type of selecting a timeslot of the words, and the location of the path of this gesture on the display device can be used to determine time and/or order of utterance of the specific words.
  • different shapes of gestures like flick, circle, drag selection and trace of the word path can be mapped to the different types of constraints and the location of the gesture on the display device can be used to determine the value of the constraint for subsequent rescoring of the possible interpretations of the speech.
  • one embodiment of the invention discloses a method for recognizing speech including a sequence of words.
  • the method includes
  • the steps of the method are performed by a processor.
  • Another embodiment discloses a system for recognizing speech including a sequence of words including a display device for displaying a set of interpretations of the speech, and for receiving, in response to the displaying, a gesture with respect to the display device; and at least one processor for determining a type and a value of a word sequence constraint based on a shape of the gesture and a location of the gesture relative to locations of elements from the set of
  • Figure 1A is a schematic of components of a system for speech recognition according to one embodiment of the invention.
  • Figure IB is a partial front view of an instrumental panel of a vehicle including the system according some embodiments of the invention.
  • Figure 2A is a block diagram illustrating the functionality of a speech recognition module according to one embodiment of the invention.
  • Figure 2B is an example of steps of the method of Figure 2 A.
  • Figure 3 A is a flowchart of the steps performed by methods for error correction in speech recognition according to various embodiments of the invention.
  • FIG. 3B is a flowchart of the steps performed by methods for error correction in speech recognition according to various embodiments of the invention.
  • FIG. 3C is a flowchart of the steps performed by methods for error correction in speech recognition according to various embodiments of the invention.
  • Figure 3C is a flowchart of the steps performed by methods for error correction in speech recognition according to various embodiments of the invention.
  • Figure 3D is a flowchart of the steps performed by methods for error correction in speech recognition according to various embodiments of the invention.
  • Figure 4 is a schematic of a sequence of words with an emphasized word according to one embodiment of the invention.
  • Figure 5 is a schematic a sequence of words with two emphasized words according to one embodiment of the invention.
  • Figure 6 is a schematic a sequence of words and a sorted set of word candidates according to one embodiment of the invention.
  • Figure 7A is a schematic of exemplar embodiments of the invention.
  • Figure 7B is a schematic of exemplar embodiments of the invention.
  • Figure 7C is a schematic of exemplar embodiments of the invention.
  • Figure 8 is a block diagram of a method for recognizing speech including a sequence of words according to one embodiment of the invention.
  • Figure 9A is a diagram of an exemplar shape of a gesture represented by a sequence of coordinate locations in the space of the input device.
  • Figure 9B is a diagram of a location of a gesture with respect to a display device represented by the location of the gesture in the coordinate space of the display device.
  • Figure 9C is an example of human machine interfaces for displaying and specifying the word-sequence constraints according to some embodiments of the invention.
  • Figure 9D is an example of human machine interfaces for displaying and specifying the word-sequence constraints according to some embodiments of the invention.
  • Figure 10 is an example of the mapping between a set of shapes of gestures and a set of types of word sequence constraint according to some embodiments of the invention.
  • Figure 1A shows an example of a system 1 capable of implementing a method for recognizing speech while minimizing a cognitive load on a user uttering the speech according to some embodiments of the invention.
  • the system 1 includes a central processing unit (CPU) 100, which controls the operation of the entire system.
  • the system 1 interacts with a memory 101, which comprises , software related to an operating system (OS) 1010 of the system, application programs 1011 that can be executed by the CPU 100 to provide specific functionalities to a user of the system, such as dictation and error correction, and software 1012 related to speech recognition.
  • the system 1 further comprises an audio interface (I/F) 102 to receive speech, which may can be recorded by microphone 103 or received from external input 104, such as speech acquired from external systems.
  • I/F audio interface
  • the system 1 can further include one or several controllers, such as a display controller 105 for controlling the operation of a display 106, which may for instance be a Liquid Crystal Display (LCD) or other type of the displays.
  • the display 106 serves as an optical user interface of system 1 and allows for example to present sequences of words to a user of the system 1.
  • the system 1 can further be connected to an audio output controller 111 for controlling the operation of an audio output system 112, e.g., one or more speakers.
  • the system 1 can further be connected to one or more input interfaces, such as a joystick controller 107 for receiving input from a joystick 108, and a keypad controller 109 for receiving input from a keypad 110.
  • the use of the joystick and/or keypad is of exemplary nature only. Equally well, a track ball, or arrow keys may be used to implement the required functionality.
  • the display 106 can be a touchscreen display serving as an interface for receiving the inputs from the user. Furthermore, due to the ability to perform speech recognition, the system 1 may completely dispense with any non-speech related interfaces altogether.
  • the audio I/F 102, joystick controller 107, keypad controller 109 and display controller 105 are controlled by CPU 100 according to the OS 1010 and/or the application program 1011 CPU 100 is currently executing.
  • the system 1 can be embedded in an instrumental panel 150 of a vehicle 199.
  • Various controls 131-133 for controlling an operation of the system 1 can be arranged on a steering wheel 130.
  • the controls 125 can be place on a control module 120.
  • the system 1 is configured to improve the interpretation of speech of the user to reduce a number of user interactions, so that the user can concentrate on operating the vehicle.
  • FIG. 2A schematically shows the functionality of a speech recognition system 200 with improved error correction capabilities according to some embodiments of the invention.
  • the speech recognition unit 200 includes a speech recognition module 210, which can be implemented by the CPU 100 of the system 1 by executing the speech recognition software 1012 stored in the memory 101.
  • the speech recognition module 210 receives a speech 230, which is a spoken representation of a sequence of words (for instance a complete sentence).
  • the speech includes one or combination of audio signal, speech features, or frame-based acoustic scores.
  • the sequence of words typically includes multiple words, e.g., three or more words.
  • the speech recognition module is configured to perform speech recognition of the speech 230 to determine a set of interpretations that, in the optimal case, resembles the sequence of words represented by the speech. Each interpretation in the set includes a sequence of words.
  • the speech recognition module can use an acoustic model 201 and a language model 202.
  • the acoustic model 201 can for instance be stored in memory 101 of the system 1 and in one embodiment describes the conditional probabilities of sequences of acoustic features given a word sequence hypothesis, or other sequence of units representing the phonetic properties of words.
  • the acoustic model can include phonemes or some other sound units.
  • the acoustic model can include a phoneme sequence model, a model of sub-phonetic state sequences, and a Gaussian mixture model for the probability of the acoustic features given each sub-phonetic state.
  • the acoustic model can include a transformation from acoustic features to phonetic state sequence probabilities that uses, for example, a neural network.
  • the language model 202 can also be stored in the memory 101 of the system 1 and can include statistics on the probability of a sequence of words comprising at least one word to occur in a language.
  • the language model 202 can for instance be a uni-gram language model that is related to the likelihood of a single word to be used in a language, or a bi-gram language model that expresses a likelihood of two words of a language following each other. Also language models considering a larger number of subsequent words may be used, e.g. a tri-gram language model, etc.
  • the speech recognition module 210 performs speech recognition by segmenting the speech into segments that are assumed to relate to single words, and then recognizes the single words, for instance by identifying phonemes in the input speech sequence segments and by comparing the phonemes to a phoneme-to-text mapping of the language model 202.
  • the speech recognition module 210 generally identifies a set of possible interpretations for each input speech sequence segment, where each interpretation includes a sequence of words. Interpretations are also known as recognition results. Each interpretation is associated with a recognition confidence value, e.g., a score representing correctness of an interpretation in representing the sequence of words. The score expresses a confidence of speech recognition that the recognition result is correct. For each input speech segment, the speech recognition module can determine the recognition result, e.g., a word, with the largest recognition confidence value, yielding a sequence of words that is considered to represent the input speech sequence.
  • the speech recognitions of speech are further refined by taking the language model 201 into account. Then, in addition to the recognition confidence values, a probability that a set of one or more words occurs in a language is taken into account in generating a set of interpretations of the speech using a language model and a recognition vocabulary. For example, in case of a bigram language model, even when a possible recognition result has a high
  • the speech recognition module 210 may nevertheless decide for "three,” using the context, for instance “at” and “o'clock” in the intended sequence of words “at three o'clock.”
  • Figure 2B shows examples of the set of interpretations of the speech 240.
  • the set of interpretations is determined or represented as an n- best list 250 of the speech or as a word lattice 260 of the speech.
  • various embodiments determine, for each interpretation in the set 251, a score, e.g., the score 255, representing correctness of an interpretation subject to the word- sequence constraint.
  • the score can be determined by a number of ways.
  • an alternative representation of the set of interpretations is a graph known as a lattice 260, in which nodes, e.g., nodes 265 and 267, represent each hypothesized word that is identified with particular region of the speech, and occurs with a particular context.
  • nodes e.g., nodes 265 and 267
  • words and word sequences that occur in the same place in many hypotheses in the w-best list can be reduced to sharing a common sub-path in the lattice.
  • the arcs of the lattice can include the acoustic and language model scores, such that the overall score for a word sequence can be derived from the scores along a path through the lattice.
  • the arcs 261, 262, and 263 can represent a score defined by corresponding probabilities of words "don't,” “want” and "wenf ' following the word "I.”
  • the speech recognition system is formulated as a probabilistic model, and the score is based on the probability of an interpretation, or sequence of words given an observed sequence of acoustic features.
  • An interpretation can include a sequence of words, and may also identify the
  • the score in some embodiments is based on the probability of the word
  • the score can be defined as the logarithm of this quantity:
  • the score is separated into different sub-scores:
  • the language model score can be computed in many ways, including using a discrete probabilistic model, a neural network, discriminatively trained conditional random fields, and so on. For example, one embodiment formulate the
  • n— 1 The probability is conditional on words occurring earlier in the sequence, known as the context.
  • the general model is too complex, and a simpler model known as an n-gram model is used, where the context is limited to the preceding n— 1 terms:
  • various other parameterizations and estimation procedures are used with n-gram models to improve their ability to generalize from training data to test data.
  • Inference in a speech recognition model can be characterized as a search over hypotheses find the best scoring hypothesis
  • the output of the search over the hypotheses can be an n-best list 250, comprising n interpretations of the sequences of words, along with the scores of the interpretations 255, e.g., one or combination of the scores of the acoustic model and the scores of the language model.
  • Some embodiments provide as an output the best scoring alignment of the words with the temporal segments of the acoustic signal.
  • the output of the search over hypotheses can be the lattice 260. Determining scores is typically more efficient in the lattice than using an n-best list. However for the clarity purposes, some embodiments are described using the n-best representation.
  • the produced recognition may still have ambiguities and errors.
  • the interpretation 251 having the highest score can still be inaccurate.
  • requesting the user to correct the specific interpretation 251 disregards other interpretations, which might include the correct interpretation.
  • the request to correct the specific interpretation can require multiple interactions with the user, and can distract the user from other tasks.
  • Some embodiments of the invention are based on a general realization that corrections of recognition results of large-vocabulary continuous speech including a sequence of words should be based on constraints for the interpretation of the entire speech. Such constraints can improve probability of interpreting the entire speech correctly, as contrasted with corrections that only modify a single word.
  • the speech recognition unit 200 also includes an error correction module 220 which reinterprets recognition results by. taking the language model 202 and a word sequence constraint 270 into account. Specifically, in some embodiments, the error correction module 220 updates the score of each interpretation based on a consistency of the interpretation with the constraint 270. After the update, the interpretation 215, e.g., with the highest score, can be determined and outputted as the recognized speech.
  • Some embodiments are based on a realization, the constraints for
  • interpreting the entire speech can be complemented by word-sequence constraints 270 of the speech provided, e.g., by a user uttering the speech.
  • the word-sequence constraints can include a number and order of the words in the speech, a specific word uttered at specific time in the speech, the absence of the specific word in the speech, a connection or separation of two specific words in the speech.
  • the word-sequence constraints can also specify the topic of the speech output, for instance that the topic of the speech is pollution.
  • the constraint 270 can be used to update the score of the entire set of the interpretation.
  • the word-sequence constraint is used
  • Figure 3 A shows a flow chart of a method for recognizing a speech of a user according to one embodiment of the invention.
  • the method 300 employs some principles and realizations described above and can be implemented using a processor 301.
  • the method receives 305 the speech representing a sequence of words and generates 310 a set of interpretations of the speech using an acoustic model and a language model.
  • the set of interpretations comprises one or more interpretations where each interpretation is a possible sequence of words that could have been spoken.
  • Various embodiments receive or generate the interpretations as an n-best list of interpretations, a word-lattice, or other representation of the set of possible word sequences.
  • the method determines 315 a correctness score for that interpretation to produce a set of scores.
  • the scores can be determined based on probabilities given by the acoustic model and the language model and/or a correctness function described below.
  • the method determines 320 the interpretations with the largest score, and
  • Various embodiments use user interface components to allow the user to provide constraints for the set of interpretations if necessary.
  • the method decides 330 whether a correction using the word sequence constraints is desired based on input from the user given through the user interface components.
  • User can provide constrains using any interface of the system 1 or a combination thereof. For example, user can use the keypad 110, the joystick 108, the touchscreen of the display 106, a speech interface 103, and combination thereof.
  • the method receives 335 a constraint on the word sequence.
  • Various embodiments receive or deteraiine the word sequence constraint based on user interface actions through user interface components such as choice lists, buttons, etc.
  • the word sequence constraints are represented as probabilities of the occurrence of various events.
  • the word sequence constraint includes metadata of the speech, such as a number of words in the speech or absence of a specific word in the speech.
  • the metadata of the speech is information about the word sequence as contrasted with the information related to the context of the word sequence. Such word-sequence constraints can be collected at any time.
  • the method updates 340 the scores of the set of interpretations of the speech according to how consistent each interpretation is with the constraint. After the scores are updated, the highest score is determined 320, and the method continues iteratively until no more corrections are desired by the user.
  • interpretations may be output from the method, e.g., stored into the memory, for being use later.
  • Figure 3B shows a flowchart of a method for updating the score of the interpretation subject to constraints.
  • the word sequence constraints are received 350 from the user.
  • the word sequence constraint is modified to determine 360 the constraints suitable for the interpretation. For example, some embodiments determine the constraints based on the word sequence constraint received from the user.
  • the word sequence constraint is received from the user in response to communicating the subset of interpretations.
  • the embodiments determine 361 a type of the constraint based on the word sequence constraint and determine 360 the constraint based on the type.
  • the type of the constraint can be one or a combination of a language type: a constraint on the language probabilities, an acoustic type: a constraint on the probability distribution of the acoustic features given the hypothesized words, and a context type: a constraint on the word sequences within the interpretations.
  • Figure 3C shows an example of determining the constraints according to some embodiments, where the type of the determined constraints is based on the scores to be updated. Those embodiments update 365 one or combination of the language and/or acoustic models. Some embodiments determine the type of the constraint based on the word sequence constraint received from the user. For example, in some embodiments, the word-sequence constraints include metadata of the speech, such as a number of words in the speech, topic of the speech, a number of words spoken within specific period of speech. Those constraints are usually language or acoustic type constraints.
  • the word-sequence constraints are more specific to the context of the speech.
  • the word-sequence constraints can include information about presence or the absence of a specific word in the speech. Those constraints are usually the context type constraints.
  • the updating of the scores is done by re-computing the language model scores, for example, using .
  • the acoustic model scores, and the estimated regions R corresponding to each word are left unchanged.
  • the type of the constraint is the language type and the updating of the scores is done by re-computing the language model scores using a modified language model. For example, instead of the original language model the lattice is rescored using .
  • the score can thus be updated by evaluating
  • One embodiment imposes a constraint that the topic is "pollution,” in a similar way.
  • the embodiment uses a constrained language model with
  • rescoring amounts to evaluating the score using the modified score function on each entry of the n-best list.
  • a graph of possible word sequences based on the lattice or n- best list, can be efficiently searched using various algorithms such as the forward- backward algorithm, Viterbi decoding, or stack decoding, using the modified score function.
  • a modified «-best list or lattice is generated.
  • the acoustic model scores such as in the above example, and the estimated regions R
  • the type of the constraint is the acoustic type and the acoustic scores are changed depending on the constraints. For example, in the case
  • the rescoring may also be dependent on the alignment.
  • One embodiment defines the model and updates the scores
  • one such constraint can be that there is only one word within a particular time region.
  • Figure 3D shows a flowchart of a method for determining absolute or relative constraints for updating the scores.
  • a constraint is absolute
  • the interpretation of the word sequence that does not match the constraint is rescored with a smallest possible score.
  • a constraint is relative, e.g:, a probabilistic
  • an interpretation that does not match the constraint is rescored with a smaller weight than interpretations that do match the constraint, but a non-matching interpretation may still have a final score higher than the scores of the interpretation consistent with the relative constraint.
  • the relative constraints have continuous values representing different degrees of constraint satisfaction.
  • Such constraints can be formulated using a multiplicative factor representing the constraint.
  • the constraint includes a constraint factor 369 the output of which is a number indicating the degree of constraint satisfaction for one or more of an acoustic sequence, a word sequence, and an alignment.
  • mis constraint factor one embodiment formulates a general re-scoring function:
  • the constraint factor can be determined using an indicator function 363 mapping the word sequence constraint specified by the user with the constraint suitable for updating the scores of the interpretation and a weight parameter 367 indicating a degree of the relative constraint (up to making the relative constraints absolute).
  • R and R in this case, and are a non-negative weight parameters representing the degree of constraint satisfaction, and, for example,
  • a constraint on the alignment can be implemented using where is an indicator function for
  • multiple constraints for the interpretation can be determined based on the word sequence constraint received from the user. For example, if the user indicated that a word in the word sequence is correct, some embodiments determine that the words at the left or at the right of the word (depending on the language model) are correct as well. In those embodiments, additional constraints for neighboring words are generated.
  • the scoring function tests for presence of the specific word.
  • a direction of the speech can be determine based on the language model, and the scoring function is updated with a test for presence of the words preceding and following, according to the direction of the speech, the specific word in the interpretation communicated to the user.
  • the scoring function test not only for the specific word, but also for other words neighboring the word from the direction of the speech.
  • the scores of some interpretations are updated 370 based on a consistency of a specific interpretation with the constraint.
  • each interpretation is updated, such that the score of each interpretation may be either increased or decreased. That means that after the update 370, the score of each interpretation may be either increased or decreased.
  • the interpretations with the largest score are determined 380, and the output to the user is formed 390 and communicated to the user.
  • Some embodiments communicate the interpretations using visual means such as words and characters shown on a display.
  • Some embodiments communicate the interpretations using audio feedback such as sounds produced from a speaker using Text-to-speech or recorded audio waveforms.
  • Various embodiments may communicate one or multiple interpretations with the best scores.
  • Figures 4, 5 and 6 show examples of interfaces for selecting the word- sequence constraint based on interpretations communicated to the user of the speech.
  • the interpretation can be an initial interpretation of the speech, or subsequent interpretations chosen after updating the scores.
  • the sequence of words 4 is a sentence that comprises five words, wherein the third word (Word3) is furnished with an emphasizing dashed frame 40.
  • the Word3 is determined as a most efficient constraint for the word-sequence.
  • the dashed frame 40 is not only meant for emphasizing, but also represents a selector, which can be moved by a user in order to select words that are erroneously recognized and thus need correction. Such movement may for instance be performed word-by-word by means of a joystick or by arrow keys, and selection then may be performed by pressing a dedicated button or key (which may for instance be integrated into or implemented by said joystick or touch screen).
  • Figure 5 shows an alternative interface, where user can select one or multiple words, e.g., Word3 and Word5.
  • Word3 and Word5 After selecting a word or word sequence, the user can perform one of the following actions: marking the selected word or sequence as correct, marking one or more of the selected word or sequence as incorrect, skipping the selected word or sequence, requesting additional choices for the selected word or sequence, or possibly altering some other constraints such as the subject matter or style of the text.
  • the time or time window associated with the selected word can be used to retrieve the set of words that could have been spoken at that time or time window, and some or all of these are displayed to the user, preferably ranked in order of decreasing scores.
  • Figure 6 exemplifies a scroll-down list 60 for the third word (Word3) of a sequence of words 6.
  • the scroll-down list 60 includes four word candidates, selected based on the time window corresponding to the word3.
  • a selector 61 is automatically placed on the first entry in said scroll-down list 60, which can be vertically moved to selected entries from said scroll-down list 60.
  • Figure 7A shows a schematic of an exemplar method for interpreting speech 710 subject to word sequence constraint according to one embodiment of the invention.
  • the word-sequence constraint includes a constraint 714 on a number of words, i.e., that the speech has six words.
  • the constraint 714 is used to update the score 712 of the entire set of interpretations 711 to produce an updated set 715 with updated scores 716 of the interpretations.
  • the scores 716 for the entire set of the interpretations are updated.
  • Figure 7B shows a schematic of an exemplar method for interpreting speech 720 subject to word sequence constraint according to one embodiment of the invention.
  • the word-sequence constraint includes a constraint 724 on a topic of the speech, i.e., that the speech is about the pollution.
  • the constraint 724 is used to update the score 722 of the entire set of the interpretation 721 to produce an updated set 725 with updated scores 716 of the interpretations.
  • Figure 7C shows a schematic of an exemplar method for interpreting speech 730 subject to word sequence constraint according to one embodiment of the invention.
  • the word-sequence constraint includes a constraint 734 on a presence of the specific word "Carts" in the speech.
  • the constraint 734 is used to update the score 732 of the entire set of the interpretation 731 to produce an updated set 735 with updated scores 736 of the interpretations.
  • Some embodiments are also based on recognition that in order for the system to impose a low cognitive load on a user, the method(s) of entry of word sequence constraints should minimize the actions of the user required for selection of the constraints.
  • the word-sequence constraints can have different values, but also can have different types.
  • the types of the word-sequence constraints can include selection or rejection of a word, selection of the sequence of word, selection of a timeslot, a trace of correct words in the sequence, replacement of words or order of words, etc. Accordingly, it is desired to provide an interface adapted for selection of both type and value of the constraints with minimal destruction of the users from their primary task, such as driving.
  • Some embodiments of the invention are based on a realization that a single gesture of the user can be analyzed to determine both the type and the value of the word-sequence constraints.
  • a type of the gesture can be used to determine a type of the word-sequence constraint
  • a path of the gesture with respect to the interface e.g., a location of the path on a display device, can be used to determine the value of the word-sequence constraint.
  • Figure 8 shows a block diagram of a method for recognizing speech including a sequence of words according to one embodiment of the invention.
  • the embodiment determines 810 a shape 817 of a gesture and a location 815 of the gesture with respect to a display device showing a set of interpretations of the speech.
  • Figure 9A shows an example of a shape 901 of a gesture made by a user on an input device.
  • the input device can represent the path of the gesture as a sequence of location coordinates 904 within the coordinate space of the input device.
  • the coordinate sequence is within a 2D plane of the touch surface.
  • the coordinate sequence is within a 3D space of motion as measured by the device.
  • the shape of the gesture is formed by the locations of these coordinates relative to each other and can include geometric patterns that are minimally affected by the transformations such as scaling and/or rotation.
  • the sequence of location coordinates is ordered and the shape of the gesture includes dynamics of formation of the gesture, such as in the grab gesture or zoom in or zoom out gestures.
  • Figure 9B shows a location 903 of the gesture having the shape 901 on a display device 902.
  • the location of the gesture can be absolute or relative.
  • the absolute location can be determined as a location of a geometrical center of the gesture, or a location of a circle circumscribing the shape of the gesture.
  • the location of the gesture can be mapped into the coordinate space of the display of UI elements and can be determined relative to the elements displayed on the UI of the input device.
  • the location 903 can be determined relative to the set of interpretations displayed on the device 902.
  • the location of the gesture is determined as the closest to the element 905.
  • the location of the gesture includes a set of locations of geometrically significant points of the shape of the gestures, such as corners of the shape corresponding to a change of the direction of the gesture.
  • the significant points can include a point closest to a particular UI element, i.e., the point of the gesture closest to a word of the interpretation of the speech.
  • the embodiment determines 850 a type of the word sequence constraint 860 based on the shape of the gesture and a mapping 830 between a set of shapes of gestures and a set of types of word sequence constraint.
  • the mapping 830 is predetermined and stored in a memory, such as the memory 101. Additionally or alternatively, the mapping 830 can be determined or updated during the operation of the speech recognition system.
  • the embodiment also determines 840 a value 845 of the word sequence constraint 860 based on the location of the gesture 815 and the shape of the gesture 817 and recognizes 820 the speech using the word sequence constraint 860. For example, a horizontal gesture can be mapped to the type of the word-sequence constraint of selecting a sequence of words. The location of the path of this gesture on the input device is mapped into a location in the coordinate space of the user interface, which can be used to determine the sequence of words present in the interpretation of the speech.
  • the vertical gesture can be mapped to the type of selecting a timeslot of the words, and the location of the path of this gesture on the input device can be mapped into a corresponding location of the gesture in user interface space, which can be used to determine time and/or order of utterance of the specific words.
  • Both the shape and location of the gesture can be represented as a single best choice, or as a probabilistic list of possibilities of shapes and locations associated with scores.
  • the shape and the location of the gesture can be updated 816 based on probability of their joint correspondence. For example, if the shape of the gesture can be interpreted as remove a word or select a sequence of words, but the location of the gesture related to a single word only, the probability of the shape of the gesture corresponding to the removing a word can be increased.
  • FIG. 9C shows an exemplar human machine interface (HMI) shown for a display device 902 for displaying and specifying the word-sequence constraints according to one embodiment of the invention.
  • HMI human machine interface
  • FIG. 9C shows an exemplar human machine interface (HMI) shown for a display device 902 for displaying and specifying the word-sequence constraints according to one embodiment of the invention.
  • words that have relatively high current correctness scores according to the speech input and the current word-sequence constraints are shown in a multidimensional
  • the interface element 910 can represent a word, a character, a phoneme, or a word sequence.
  • the horizontal dimension represents time and the vertical dimension represents the score of each element, with elements having a higher score towards the top of the visualization.
  • Figure 8 shows several kinds of selections that can be made in order to specify the word sequence constraint.
  • a single word selection 930 in the word sequence can be shown to the user, e.g, by highlighting the interface element using color, background shading or other visual information.
  • a word sequence selection 940 selects a sequence of words.
  • a time-slice selection 920 selects a time bounded section of the utterance and refers to all possible words that could have been said during that time section.
  • the HMI can be implemented using different kinds of display devices 902 such as a heads-up or wearable display with a gesture input device, or a display screen with direct touch such as a touchscreen. Other combinations of visual output and touch, mouse or gesture input are possible.
  • the embodiment maps particular gesture shapes either to particular kinds of constraints, or to actions to select a word 830, a word sequence 840, and/or a time-slice 820. After a particular element is selected, further interface actions impose word-sequence constraints upon the selected element or elements.
  • some gestures may be segmented into separate gesture segments which may be analyzed separately for shape and location information.
  • Figure 9D shows an example of a specific type of action that allows the user to prefer one particular interpretation of the utterance and to provide the word sequence constraint for that interpretation.
  • the display device 902 provides a user with an ability to trace a curve through the interface elements representing correct words.
  • Figure 9D shows examples of the two different traces.
  • the trace 950 represents the interpretation "I want to get some rice and beans," and the trace 960 represents the interpretation "I went to get the same jeans.”
  • Gesture and touch traces are both ambiguous and noisy, so the touch/gesture trace can be converted to a probability where Z is the path of the
  • V is the layout of the visual elements on the display device
  • W is the word sequence in the interpretation.
  • some embodiments determine a constraint factor indicating a degree of consistency with the word sequence constraint.
  • the constraint factor can be used for determining a scoring function based on the constraint factor and updating the set of scores using the scoring function.
  • Figure 10 shows an example of the mapping 830 between a set of types of gestures 1020 and a set of types of word sequence constraint 1010.
  • the location of the touch/gesture action provides context about what is to be selected or where the constraint is to be imposed. For example, making an "X" gesture over a word in the visualization V might map to rejecting a word (reducing its score to very low value).
  • the types of the word sequence constraint determine the types of the constraint factor. Those embodiments determining the constraint factor based on the type of the word sequence constraint. For example, in one embodiment the mapping 830 includes the gesture of a horizontal selection, and corresponding the constraint factor outputs a number indicating a degree of consistency with a word sequence. In another embodiment the mapping 830 includes the gesture of a trace of a correct word sequence, and corresponding the constraint factor outputs a number indicating a degree of consistency with the trace.
  • a constraint factor outputting a number indicating a degree of consistency of a word sequence with the trace and visualization. For example, may be
  • consistency of a word sequence with the gesture and visualization may be proportional to
  • the mapping includes one or combination of a select constraint corresponding to a grab gesture, a sequence constraint corresponding to a horizontal gesture, a synonyms constraint corresponding to a circle gesture, a trace constraint corresponding to a trace a path gesture, an intersection constraint corresponding to a hat gesture, a reorder constraint corresponding to a trace a drag gesture.
  • the embodiments can be implemented in any of numerous ways.
  • the embodiments may be implemented using hardware, software or a combination thereof.
  • the software code can be executed on any suitable processor or collection of processors, whether provided in a single computer or distributed among multiple computers.
  • processors may be implemented as integrated circuits, with one or more processors in an integrated circuit component.
  • a processor may be implemented using circuitry in any suitable format.
  • a computer may be embodied in any of a number of forms, such as a rack-mounted computer, a desktop computer, a laptop computer, minicomputer, or a tablet computer.
  • a computer may have one or more input and output systems. These systems can be used, among other things, to present a user interface.
  • Such computers may be interconnected by one or more networks in any suitable form, including as a local area network or a wide area network, such as an enterprise network or the Internet.
  • networks may be based on any suitable technology and may operate according to any suitable protocol and may include wireless networks, wired networks or fiber optic networks.
  • the various methods or processes outlined herein may be coded as software that is executable on one or more processors that employ any one of a variety of operating systems or platforms. Additionally, such software may be written using any of a number of suitable programming languages and/or programming or scripting tools, and also may be compiled as executable machine language code or intermediate code that is executed on a framework or virtual machine.
  • program or "software” are used herein in a generic sense to refer to any type of computer code or set of computer-executable instructions that can be employed to program a computer or other processor to implement various aspects of the present invention as discussed above.
  • Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other systems.
  • program modules include routines, programs, objects, components, and data structures that perform particular tasks or implement particular abstract data types.
  • functionality of the program modules may be combined or
  • embodiments of the invention may be embodied as a method, of which an example has been provided.
  • the acts performed as part of the method may be ordered in any suitable way. Accordingly, embodiments may be constructed in which acts are performed in an order different than illustrated, which may include performing some acts simultaneously, even though shown as sequential acts in illustrative embodiments.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Acoustics & Sound (AREA)
  • Computational Linguistics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Artificial Intelligence (AREA)
  • General Health & Medical Sciences (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • Machine Translation (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

Ce procédé permettant de reconnaître un discours comprenant une séquence de mots consiste à déterminer une forme d'un geste et un emplacement du geste par rapport à un dispositif d'affichage indiquant un ensemble d'interprétations du discours. Le procédé détermine un type de contrainte de séquence de mots d'après la forme du geste et détermine une valeur de la contrainte de séquence de mots d'après l'emplacement du geste. Ensuite, le discours est reconnu au moyen de la contrainte de séquence de mots.
PCT/JP2015/071426 2014-07-22 2015-07-22 Procédé et système permettant de reconnaître un discours comprenant une séquence de mots WO2016013685A1 (fr)

Priority Applications (3)

Application Number Priority Date Filing Date Title
DE112015003357.2T DE112015003357B4 (de) 2014-07-22 2015-07-22 Verfahren und System zum Erkennen einer eine Wortabfolge enthaltenden Sprachansage
JP2016554899A JP6272496B2 (ja) 2014-07-22 2015-07-22 単語のシーケンスを含む音声を認識するための方法及びシステム
CN201580040168.8A CN106537489B (zh) 2014-07-22 2015-07-22 用于识别包括单词序列的语音的方法和系统

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US14/337,472 US9196246B2 (en) 2013-06-14 2014-07-22 Determining word sequence constraints for low cognitive speech recognition
US14/337472 2014-07-22

Publications (1)

Publication Number Publication Date
WO2016013685A1 true WO2016013685A1 (fr) 2016-01-28

Family

ID=54105951

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2015/071426 WO2016013685A1 (fr) 2014-07-22 2015-07-22 Procédé et système permettant de reconnaître un discours comprenant une séquence de mots

Country Status (4)

Country Link
JP (1) JP6272496B2 (fr)
CN (1) CN106537489B (fr)
DE (1) DE112015003357B4 (fr)
WO (1) WO2016013685A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018044887A1 (fr) * 2016-08-31 2018-03-08 Nuance Communications, Inc. Interface utilisateur destinée à une application de dictée mettant en œuvre une reconnaissance vocale automatique
US10515626B2 (en) 2016-03-23 2019-12-24 Google Llc Adaptive audio enhancement for multichannel speech recognition

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108871370A (zh) * 2018-07-03 2018-11-23 北京百度网讯科技有限公司 导航方法、装置、设备和介质
CN113168829A (zh) * 2018-12-03 2021-07-23 谷歌有限责任公司 语音输入处理

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5960447A (en) 1995-11-13 1999-09-28 Holt; Douglas Word tagging and editing system for speech recognition
US5970460A (en) 1997-12-05 1999-10-19 Lernout & Hauspie Speech Products N.V. Speech recognition and editing system
US6064959A (en) 1997-03-28 2000-05-16 Dragon Systems, Inc. Error correction in speech recognition
WO2005119642A2 (fr) * 2004-06-02 2005-12-15 America Online, Incorporated Desambiguisation multimodale de reconnaissance vocale
US20060293889A1 (en) 2005-06-27 2006-12-28 Nokia Corporation Error correction for speech recognition systems
US7444286B2 (en) 2001-09-05 2008-10-28 Roth Daniel L Speech recognition using re-utterance recognition

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4604178B2 (ja) * 2004-11-22 2010-12-22 独立行政法人産業技術総合研究所 音声認識装置及び方法ならびにプログラム
JP5430382B2 (ja) * 2009-12-16 2014-02-26 キヤノン株式会社 入力装置及び方法
US20120226498A1 (en) * 2011-03-02 2012-09-06 Microsoft Corporation Motion-based voice activity detection
JP5621681B2 (ja) * 2011-03-29 2014-11-12 株式会社デンソー 車載用情報提示装置
US9092394B2 (en) * 2012-06-15 2015-07-28 Honda Motor Co., Ltd. Depth based context identification
WO2014041607A1 (fr) * 2012-09-11 2014-03-20 株式会社東芝 Dispositif de traitement d'informations, procédé de traitement d'informations, et programme
US9182826B2 (en) * 2012-11-21 2015-11-10 Intel Corporation Gesture-augmented speech recognition
JP2014149612A (ja) * 2013-01-31 2014-08-21 Nippon Hoso Kyokai <Nhk> 音声認識誤り修正装置およびそのプログラム
CN203366349U (zh) * 2013-07-19 2013-12-25 中国农业银行股份有限公司镇江分行 银行客户的身份识别系统
CN103645876B (zh) * 2013-12-06 2017-01-18 百度在线网络技术（北京）有限公司 语音输入方法和装置

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5960447A (en) 1995-11-13 1999-09-28 Holt; Douglas Word tagging and editing system for speech recognition
US6064959A (en) 1997-03-28 2000-05-16 Dragon Systems, Inc. Error correction in speech recognition
US5970460A (en) 1997-12-05 1999-10-19 Lernout & Hauspie Speech Products N.V. Speech recognition and editing system
US7444286B2 (en) 2001-09-05 2008-10-28 Roth Daniel L Speech recognition using re-utterance recognition
WO2005119642A2 (fr) * 2004-06-02 2005-12-15 America Online, Incorporated Desambiguisation multimodale de reconnaissance vocale
US20060293889A1 (en) 2005-06-27 2006-12-28 Nokia Corporation Error correction for speech recognition systems

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
WU-CHUN FENG: "An integrated multimedia environment for speech recognition using handwriting and written gestures", SYSTEM SCIENCES, 2003. PROCEEDINGS OF THE 36TH ANNUAL HAWAII INTERNATI ONAL CONFERENCE ON 6-9 JAN. 2003, PISCATAWAY, NJ, USA,IEEE, 6 January 2003 (2003-01-06), pages 128 - 137, XP010626438, ISBN: 978-0-7695-1874-9 *

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10515626B2 (en) 2016-03-23 2019-12-24 Google Llc Adaptive audio enhancement for multichannel speech recognition
US11257485B2 (en) 2016-03-23 2022-02-22 Google Llc Adaptive audio enhancement for multichannel speech recognition
US11756534B2 (en) 2016-03-23 2023-09-12 Google Llc Adaptive audio enhancement for multichannel speech recognition
WO2018044887A1 (fr) * 2016-08-31 2018-03-08 Nuance Communications, Inc. Interface utilisateur destinée à une application de dictée mettant en œuvre une reconnaissance vocale automatique
US10706210B2 (en) 2016-08-31 2020-07-07 Nuance Communications, Inc. User interface for dictation application employing automatic speech recognition

Also Published As

Publication number Publication date
JP2017511899A (ja) 2017-04-27
CN106537489B (zh) 2020-01-17
DE112015003357T5 (de) 2017-04-06
JP6272496B2 (ja) 2018-01-31
DE112015003357B4 (de) 2021-01-14
CN106537489A (zh) 2017-03-22

Similar Documents

Publication Publication Date Title
US9196246B2 (en) Determining word sequence constraints for low cognitive speech recognition
US9159317B2 (en) System and method for recognizing speech
JP6743300B2 (ja) ニューラルネットワークを用いてハイブリッド音声認識結果をランク付けするためのシステム及び方法
US10551915B2 (en) Gaze based text input systems and methods
JP3943492B2 (ja) ディクテーションとコマンドの区別を向上させる方法
US9361883B2 (en) Dictation with incremental recognition of speech
US20090228273A1 (en) Handwriting-based user interface for correction of speech recognition errors
US20080177541A1 (en) Voice recognition device, voice recognition method, and voice recognition program
US20100241431A1 (en) System and Method for Multi-Modal Input Synchronization and Disambiguation
EP3152754B1 (fr) Modification de contenu visuel pour faciliter une meilleure reconnaissance de la parole
WO2016013685A1 (fr) Procédé et système permettant de reconnaître un discours comprenant une séquence de mots
KR20210108557A (ko) 음성 인식률의 향상을 위한 음성 인식 지원 방법 및 시스템
JP2006189730A (ja) 音声対話方法および音声対話装置
CN105786204A (zh) 信息处理方法和电子设备
KR102333029B1 (ko) 발음 평가 방법 및 이를 이용한 디바이스
US20200272840A1 (en) Method and device for displaying handwriting-based entry
KR20140086302A (ko) 음성과 제스처를 이용한 명령어 인식 장치 및 그 방법
CN113990351A (zh) 纠音方法、纠音装置及非瞬时性存储介质
JP4565768B2 (ja) 音声認識装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15763100

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2016554899

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 112015003357

Country of ref document: DE

122 Ep: pct application non-entry in european phase

Ref document number: 15763100

Country of ref document: EP

Kind code of ref document: A1