EP2329491B1 - Hybridspracherkennung - Google Patents

Hybridspracherkennung Download PDF

Info

Publication number
EP2329491B1
EP2329491B1 EP09810709.7A EP09810709A EP2329491B1 EP 2329491 B1 EP2329491 B1 EP 2329491B1 EP 09810709 A EP09810709 A EP 09810709A EP 2329491 B1 EP2329491 B1 EP 2329491B1
Authority
EP
European Patent Office
Prior art keywords
speech recognition
results
speech
engine
recognizer
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP09810709.7A
Other languages
English (en)
French (fr)
Other versions
EP2329491A2 (de
EP2329491A4 (de
Inventor
Detlef Koll
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
MULTIMODAL TECHNOLOGIES LLC
Original Assignee
Multimodal Technologies LLC
MULTIMODAL Tech LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Multimodal Technologies LLC, MULTIMODAL Tech LLC filed Critical Multimodal Technologies LLC
Publication of EP2329491A2 publication Critical patent/EP2329491A2/de
Publication of EP2329491A4 publication Critical patent/EP2329491A4/de
Application granted granted Critical
Publication of EP2329491B1 publication Critical patent/EP2329491B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/28—Constructional details of speech recognition systems
    • G10L15/30—Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/28—Constructional details of speech recognition systems
    • G10L15/32—Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems

Definitions

  • ASRs automatic speech recognizers
  • Some applications of automatic speech recognizers require shorter turnaround times (the amount of time between when the speech is spoken and when the speech recognizer produces output) than others in order to appear responsive to the end user.
  • a speech recognizer that is used for a "live" speech recognition application such as controlling the movement of an on-screen cursor, may require a shorter turnaround time (also referred to as a "response time") than a speech recognizer that is used to produce a transcript of a medical report.
  • the desired turnaround time may depend, for example, on the content of the speech utterance that is processed by the speech recognizer. For example, for a short command-and-control utterance, such as "close window,” a turnaround time above 500ms may appear sluggish to the end user. In contrast, for a long dictated sentence which the user desires to transcribe into text, response times of 1000ms may be acceptable to the end user. In fact, in the latter case users may prefer longer response times because they may otherwise feel that their speech is being interrupted by the immediate display of text in response to their speech. For longer dictated passages, such as entire paragraphs, even longer response times of multiple seconds may be acceptable to the end user.
  • One known technique for overcoming these resource constraints in the context of embedded devices is to delegate some or all of the speech recognition processing responsibility to a speech recognition server that is located remotely from the embedded device and which has significantly greater computing resources than the embedded device.
  • the embedded device When a user speaks into the embedded device in this situation, the embedded device does not attempt to recognize the speech using its own computing resources. Instead, the embedded device transmits the speech (or a processed form of it) over a network connection to the speech recognition server, which recognizes the speech using its greater computing resources and therefore produces recognition results more quickly than the embedded device could have produced with the same accuracy.
  • the speech recognition server then transmits the results back over the network connection to the embedded device.
  • this technique produces highly-accurate speech recognition results more quickly than would otherwise be possible using the embedded device alone.
  • the document EP 1 617 410 A1 discloses a client-server speech recognition system, where the client waits a predetermined amount of time for the server recognition results to arrive after a recognition process in the client has finished.
  • server-side speech recognition technique has a variety of shortcomings.
  • server-side speech recognition relies on the availability of high-speed and reliable network connections, the technique breaks down if such connections are not available when needed.
  • the potential increases in speed made possible by server-side speech recognition may be negated by use of a network connection without sufficiently high bandwidth.
  • the typical network latency of an HTTP call to a remote server can range from 100ms to 500ms. If spoken data arrives at a speech recognition server 500ms after it is spoken, it will be impossible for that server to produce results quickly enough to satisfy the minimum turnaround time (500ms) required by command-and-control applications. As a result, even the fastest speech recognition server will produce results that appear sluggish if used in combination with a slow network connection.
  • a hybrid speech recognition system uses a client-side speech recognition engine and a server-side speech recognition engine to produce speech recognition results for the same speech.
  • An arbitration engine produces speech recognition output based on one or both of the client-side and server-side speech recognition results.
  • FIG. 1 a dataflow diagram is shown of a speech recognition system 100 according to one embodiment of the present invention.
  • FIG. 2 a flowchart is shown of a method 200 performed by the system 100 of FIG. 1 according to one embodiment of the present invention.
  • a user 102 of a client device 106 speaks and thereby provides speech 104 to the client device (step 202).
  • the client device 106 may be any device, such as a desktop or laptop computer, cellular telephone, personal digital assistant (PDA), or telephone.
  • PDA personal digital assistant
  • Embodiments of the present invention are particularly useful in conjunction with resource-constrained clients, such as computers or mobile computing devices with slow processors or small amounts of memory, or computers running resource-intensive software.
  • the device 106 may receive the speech 104 from the user 102 in any way, such as through a microphone connected to a sound card.
  • the speech 104 may be embodied in an audio signal which is tangibly stored in a computer-readable medium and/or transmitted over a network connection or other channel.
  • the client device 106 includes an application 108, such as a transcription application or other application which needs to recognize the speech 104.
  • the application 108 transmits the speech 104 to a delegation engine 110 (step 204).
  • the application 108 may process the speech 104 in some way and provide the processed version of the speech 104, or other data derived from the speech 104, to the delegation engine 110.
  • the delegation engine 110 itself may process the speech 104 (in addition to or instead of any processing performed on the speech by the application) in preparation for transmitting the speech for recognition.
  • the delegation engine 110 may present the same interface to the application 108 as that presented by a conventional automatic speech recognition engine.
  • the application 108 may provide the speech 104 to the delegation engine 110 in the same way that it would provide the speech 104 directly to a conventional speech recognition engine.
  • the creator of the application 108 therefore, need not know that the delegation engine 110 is not itself a conventional speech recognition engine.
  • the delegation engine 110 also provides speech recognition results back to the application 108 in the same manner as a conventional speech recognition engine. Therefore, the delegation engine 110 appears to perform the same function as a conventional speech recognition engine from the perspective of the application 108.
  • the delegation engine 110 provides the speech 104 (or a processed form of the speech 104 or other data derived from the speech 104) to both a client-side automatic speech recognition engine 112 in the client device 106 (step 206) and to a server-side automatic speech recognition engine 120 in a server 118 located remotely over a network 116 (step 208).
  • the server 118 may be a computing device which has significantly greater computing resources than the client device.
  • the client-side speech recognizer 112 and server-side speech recognizer 120 may be conventional speech recognizers.
  • the client-side speech recognizer 112 and server-side speech recognizer 120 may, however, differ from each other.
  • the server-side speech recognizer 120 may use more complex speech recognition models which require more computing resources than those used by the client-side speech recognizer 112.
  • one of the speech recognizers 112 and 120 may be speaker-independent, while the other may be adapted to the voice of the user 102.
  • the client-side recognizer 112 and server-side recognizer 120 may have different response times due to a combination of differences in the computing resources of the client 106 and server 118, differences in the speech recognizers themselves 112 and 120, and the fact that the results from the server-side recognizer 120 must be provided back to the client device 106 over the network 116, thereby introducing latency not incurred by the client-side recognizer 112.
  • Responsibilities may be divided between the client-side speech recognizer 112 and server-side speech recognizer 120 in various ways, whether or not such recognizers 112 and 120 differ from each other.
  • the client -side speech recognizer 112 may be used solely for command-and-control speech recognition, while the server-side speech recognizer 112 may be used for both command-and-control and dictation recognition.
  • the client-side recognizer 112 may only be permitted to utilize up to a predetermined maximum percentage of processor time on the client device 106.
  • the delegation engine 110 may be configured to transmit appropriate speech to the client-side recognizer 112 and server-side recognizer 120 in accordance with the responsibilities of each.
  • the client-side recognizer 112 produces speech recognition results 114, such as text based on the speech 104 (step 210).
  • the server-side recognizer 120 produces speech recognition results 122, such as text based on the speech 104 (step 212).
  • the results 114 may include other information, such as the set of best candidate words, confidence measurements associated with those words, and other output typically provided by speech recognition engines.
  • the client-side results 114 and server-side results 122 may differ from each other.
  • the client-side recognizer 112 and server-side recognizer 120 both provide their results 114 and 112, respectively, to an arbitration engine 124 in the client device 106.
  • the arbitration engine 124 analyzes one or both of the results 114 and 122 to decide which of the two results 114 and 122 to provide (as results 126) to the delegation engine 110 (step 214).
  • the arbitration engine 124 may perform step 214 either after receiving both of the results 114 and 122, or after receiving one of the results 114 and 122 but not the other. Therefore, in general the arbitration engine 124 produces the output 126 based on the client-side results 114 and/or the server-side results 122.
  • the delegation engine 110 provides the selected results 126 back to the requesting application 108 (step 216).
  • the requesting application 108 receives speech recognition results 126 back from the delegation engine 110 as if the delegation engine 110 were a single, integrated speech recognition engine 110.
  • the details of the operations performed by the delegation engine 110 and arbitration engine 124 are hidden from the requesting application 108.
  • the arbitration engine 124 may use any of a variety techniques to select which of the client-side results 114 and server-side results 122 to provide to the delegation engine 110. For example, as illustrated by the method 300 of FIG. 3A , the arbitration engine 124 may select the client-side results 114 as soon as those results 114 become available (step 302), if the server-side recognizer 120 is not accessible over the network (e.g., if the connection between the client 106 and the network 116 is down) (steps 304-306).
  • the arbitration engine 124 may select the server-side results 122 as soon as those results 122 become available (step 312), if the client-side recognizer 112 is not accessible (steps 314-316). This may occur, for example, if the client-side recognizer 112 has been disabled as a result of a high-priority CPU task being executed on the client device 106.
  • the arbitration engine 124 may select the server-side recognizer's results 122 if those results 122 become available no later than a predetermined waiting time after the client-side recognizer's results 114 became available.
  • the arbitration engine 124 may return the server-side results 122 (step 330) only if they are received (step 324) before the predetermined waiting time has passed (step 326). If the server-side results 122 are not available by that time, then the arbitration engine 124 may return the client-side results 114 (step 328).
  • the predetermined waiting time may be selected in any way.
  • the predetermined waiting time may depend on the type of recognition result.
  • the predetermined waiting time applied by the method 320 to command-and-control grammars may be selected to be shorter than the predetermined waiting time applied to dictation grammars.
  • a predetermined waiting time of 500ms may be applied to command-and-control grammars, while a predetermined waiting time of 1000ms may be applied to dictation grammars.
  • the arbitration engine 124 may select the client-side recognizer's results 114 (step 346) as soon as those results 114 become available (step 342), if the confidence measure associated with those results 114 exceeds some predetermined threshold value (step 344).
  • the arbitration engine 124 is not limited to "selecting" one or the other of the results 114 and 122 produced by the client-side recognizer 112 and server-side recognizer 120, respectively. Rather, for example, as illustrated by the method 350 of FIG. 3E , the arbitration engine 124 may receive the results 114 and 122 (steps 352 and 354), and combine or otherwise process those results 114 and 122 in various ways (step 356) to produce the output 126 provided back to the requesting application 108 (step 358). For example, the arbitration engine 124 may combine the results 114 and 122 using a well-known technology named ROVER (Recognizer Output Voting Error Reduction), or using other techniques, to produce the output 126.
  • ROVER Recognition Output Voting Error Reduction
  • the arbitration engine 124 may combine the techniques disclosed above with respect to FIGS. 3A-3E , and with other techniques, in any combination.
  • the method 340 of FIG. 3D may be combined with the method 320 of FIG. 3C by performing steps 344 and 346 of method 340 after step 322 in FIG. 3C , and proceeding to step 324 of FIG. 3C if the confidence measure in step 344 does not exceed the threshold.
  • results from one of the recognizers 112 and 120 may overlap in time with the results from the other recognizer, as illustrated by the method 400 of FIG. 4A .
  • the speech 104 is five seconds in duration
  • the client-side recognizer 112 produces high-confidence results 114 for the first two seconds of the speech 104 (step 402).
  • the arbitration engine 124 may submit those results 114 to the delegation engine 110, which commits those results 114 (i.e., includes the results 114 in the results 126 that are passed back to the application 108) before the server-side results 122 become available (step 404).
  • the server-side results 122 for some or all of the same five seconds of speech 104 may conflict (overlap in time) with some or all the client-side results 114 (step 406).
  • the arbitration engine 124 may take action in response to such overlap (step 408).
  • the arbitration engine 124 may consider results 114 and 122 to be non-overlapping and process them in any of the ways described above with respect to FIGS. 3A-3E (step 414). Otherwise, the arbitration engine 124 may consider the results 114 and 122 to be overlapping and process them accordingly, such as in the ways described in the following examples (step 416).
  • some predetermined threshold time period e.g. 100ms
  • the arbitration engine 124 may consider one of the recognizers (e.g., the server-side recognizer 120) to be preferred over the other recognizer. In this case, if results (e.g., client-side results 114) from the non-preferred recognizer arrive first (step 422) and are committed first (step 424), and then results (e.g., server-side results 122) from the preferred recognizer arrive (step 428) which overlap with the previously-committed non-preferred results, the arbitration engine 124 may commit (i.e., include in the hybrid results 126) the preferred results (e.g., server-side results 122) as well (step 430). Although this results in certain portions of the speech 104 being committed twice, this may produce more desirable results than discarding the results of a preferred recognizer. If the later-received results are not from the preferred recognizer, those results may be discarded rather than committed (step 432).
  • results e.g., client-side results 114 from the non-preferred recognizer arrive first (step 422) and are committed
  • results e.g., server-side results 122 from the preferred recognizer arrive first (step 442) and are committed first (step 444), and then results (e.g., client-side results 114) from the non-preferred recognizer arrive which overlap with the previously-committed preferred results (steps 446 and 448), then the arbitration engine 124 may discard the non-preferred results (step 450). Otherwise, the arbitration engine 124 may commit the later-received results or process them in another manner (step 452).
  • step 4E (which represents one embodiment of step 408 of FIG. 4A )
  • the arbitration engine 124 may ignore the words from the new recognition results that overlap in time with the words from the old recognition results (using timestamps associated with each word in both recognition results) (step 462), and then commit the remaining (non-overlapping) words from the new recognition results (step 464).
  • the arbitration engine 124 may use the newly-received results to update the previously-committed results (step 472). For example, the arbitration engine 124 may determine whether the confidence measure associated with the newly-received results exceeds the confidence measure associated with the previously-committed results (step 474) and, if so, replace the previously-committed results with the newly-received results (step 476).
  • Embodiments of the present invention have a variety of advantages.
  • a client-side device such as a cellular telephone
  • the techniques disclosed herein effectively produce a hybrid speech recognition engine which uses both the client-side recognizer 112 and server-side recognizer 118 to produce better results than either of those recognizers could have produced individually. More specifically, the resulting hybrid result can have better operating characteristics with respect to system availability, recognition quality, and response time than could be obtained from either of the component recognizers 112 and 120 individually.
  • the techniques disclosed herein may be used to satisfy the user's turnaround time requirements even as the availability of the network 116 fluctuates over time, and even as the processing load on the CPU of the client device 106 fluctuates over time.
  • Such flexibility results from the ability of the arbitration engine 124 to respond to changes in the turnaround times of the client-side recognizer 112 and server-side recognizer 120, and in response to other time-varying factors.
  • Embodiments of the present invention thereby provide a distinct benefit over conventional server-side speech recognition techniques, which break down if the network slows down or becomes unavailable.
  • Hybrid speech recognition systems implemented in accordance with embodiments of the present invention may provide higher speech recognition accuracy than is provided by the faster of the two component recognizers (e.g., the server-side recognizer 120 in FIG. 1 ). This is a distinct advantage over conventional server-side speech recognition techniques, which only provide results having the accuracy of the server-side recognizer, since that is the only recognizer used by the system.
  • hybrid speech recognition systems implemented in accordance with embodiments of the present invention may provide a faster average response time than is provided by the slower of the two component recognizers (e.g., the client-side recognizer 112 in FIG. 1 ).
  • This is a distinct advantage over conventional server-side speech recognition techniques, which only provide results having the response time of the server-side recognizer, since that is the only recognizer used by the system.
  • Each of the client-side recognizer 112 and server-side recognizer 120 may be any kind of recognizer. Each of them may be chosen without knowledge of the characteristics of the other. Multiple client-side recognizers, possibly of different types, may be used in conjunction with a single server-side recognizer to effectively form multiple hybrid recognition systems. Either of the client-side recognizer 112 or server-side recognizer 120 may be modified or replaced without causing the hybrid system to break down. As a result, the techniques disclosed herein provide a wide degree of flexibility that makes them suitable for use in conjunction with a wide variety of client-side and server-side recognizers.
  • the techniques disclosed herein may be implemented without requiring any modification to existing applications which rely on speech recognition engines.
  • the delegation engine 110 may provide the same interface to the application 108 as a conventional speech recognition engine.
  • the application 108 may provide input to and receive output from the delegation engine 110 as if the delegation engine 110 were a conventional speech recognition engine.
  • the delegation engine 110 therefore, may be inserted into the client device 106 in place of a conventional speech recognition engine without requiring any modifications to the application 108.
  • the techniques described above may be implemented, for example, in hardware, software tangibly stored on a computer-readable medium, firmware, or any combination thereof.
  • the techniques described above may be implemented in one or more computer programs executing on a programmable computer including a processor, a storage medium readable by the processor (including, for example, volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device.
  • Program code may be applied to input entered using the input device to perform the functions described and to generate output.
  • the output may be provided to one or more output devices.
  • Each computer program within the scope of the claims below may be implemented in any programming language, such as assembly language, machine language, a high-level procedural programming language, or an object-oriented programming language.
  • the programming language may, for example, be a compiled or interpreted programming language.
  • Each such computer program may be implemented in a computer program product tangibly embodied in a machine-readable storage device for execution by a computer processor.
  • Method steps of the invention may be performed by a computer processor executing a program tangibly embodied on a computer-readable medium to perform functions of the invention by operating on input and generating output.
  • Suitable processors include, by way of example, both general and special purpose microprocessors.
  • the processor receives instructions and data from a read-only memory and/or a random access memory.
  • Storage devices suitable for tangibly embodying computer program instructions include, for example, all forms of non-volatile memory, such as semiconductor memory devices, including EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROMs. Any of the foregoing may be supplemented by, or incorporated in, specially-designed ASICs (application-specific integrated circuits) or FPGAs (Field-Programmable Gate Arrays).
  • a computer can generally also receive programs and data from a storage medium such as an internal disk (not shown) or a removable disk.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Telephonic Communication Services (AREA)
  • Computer And Data Communications (AREA)

Claims (4)

  1. Computerimplementiertes Verfahren, das durch eine Client-Einrichtung (106) ausgeführt wird, wobei das Verfahren umfasst:
    (A) Empfangen einer Anforderung von einem Anforderer, eine automatische Spracherkennung auf ein Audiosignal anzuwenden;
    (B) Bereitstellen des Audiosignals an eine erste Maschine zur automatischen Spracherkennung in der Client-Einrichtung;
    (C) Empfangen (322) von ersten Spracherkennungsergebnissen von der ersten Maschine zur automatischen Spracherkennung (112) zu einem ersten Zeitpunkt;
    (D) Bereitstellen des Audiosignals an eine zweite Maschine zur automatischen Spracherkennung (120) in einer Server-Einrichtung (118);
    (E) Bestimmen (324, 326), ob innerhalb eines vorbestimmten Zeitraums nach dem ersten Zeitpunkt zweite Spracherkennungsergebnisse durch die Client-Einrichtung von der zweiten Maschine zur automatischen Spracherkennung empfangen wurden;
    (F) wenn die zweiten Spracherkennungsergebnisse durch die Client-Einrichtung innerhalb des vorbestimmten Zeitraums empfangen wurden, Bereitstellen (330) der zweiten Spracherkennungsergebnisse an den Anforderer in Reaktion auf die Anforderung; und
    (G) wenn die zweiten Spracherkennungsergebnisse durch die Client-Einrichtung nicht innerhalb des vorbestimmten Zeitraums empfangen wurden, Bereitstellen (328) der ersten Spracherkennungsergebnisse an den Anforderer in Reaktion auf die Anforderung;
    und wobei (E) das Auswählen des vorbestimmten Zeitraums auf Basis eines Typs der zweiten Spracherkennungsergebnisse umfasst.
  2. Computerimplementiertes Verfahren nach Anspruch 1, wobei der vorbestimmte Zeitraum, der für zweite Spracherkennungsergebnisse eines Befehls- und-Steuerungsgrammatiktyps ausgewählt wird, kürzer ist als der vorbestimmte Zeitraum, der für zweite Spracherkennungsergebnisse eines Diktatgrammatiktyps ausgewählt wird.
  3. Vorrichtung, umfassend:
    Mittel zum Empfangen einer Anforderung von einem Anforderer, eine automatische Spracherkennung auf ein Audiosignal anzuwenden;
    Mittel zum Bereitstellen des Audiosignals an eine erste Maschine zur automatischen Spracherkennung (112) in der Client-Einrichtung (106);
    Mittel zum Empfangen von ersten Spracherkennungsergebnissen von der ersten Maschine zur automatischen Spracherkennung zu einem ersten Zeitpunkt;
    Mittel zum Bereitstellen des Audiosignals an eine zweite Maschine zur automatischen Spracherkennung (120) in einer Server-Einrichtung (118);
    Mittel zum Bestimmen, ob innerhalb eines vorbestimmten Zeitraums nach dem ersten Zeitpunkt zweite Spracherkennungsergebnisse durch die Client-Einrichtung von der zweiten Maschine zur automatischen Spracherkennung empfangen wurden;
    Mittel zum Bereitstellen der zweiten Spracherkennungsergebnisse an den Anforderer in Reaktion auf die Anforderung, wenn die zweiten Spracherkennungsergebnisse durch die Client-Einrichtung innerhalb des vorbestimmten Zeitraums empfangen wurden; und
    Mittel zum Bereitstellen der ersten Spracherkennungsergebnisse an den Anforderer in Reaktion auf die Anforderung, wenn die zweiten Spracherkennungsergebnisse durch die Client-Einrichtung nicht innerhalb des vorbestimmten Zeitraums empfangen wurden;
    und wobei der vorbestimmte Zeitraum auf Basis eines Typs der zweiten Spracherkennungsergebnisse ausgewählt wird.
  4. Vorrichtung nach Anspruch 3, wobei der vorbestimmte Zeitraum, der für zweite Spracherkennungsergebnisse eines Befehls- und- Steuerungsgrammatiktyps ausgewählt wird, kürzer ist als der vorbestimmte Zeitraum, der für zweite Spracherkennungsergebnisse eines Diktatgrammatiktyps ausgewählt wird.
EP09810709.7A 2008-08-29 2009-08-31 Hybridspracherkennung Active EP2329491B1 (de)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US9322008P 2008-08-29 2008-08-29
US12/550,380 US7933777B2 (en) 2008-08-29 2009-08-30 Hybrid speech recognition
PCT/US2009/055479 WO2010025440A2 (en) 2008-08-29 2009-08-31 Hybrid speech recognition

Publications (3)

Publication Number Publication Date
EP2329491A2 EP2329491A2 (de) 2011-06-08
EP2329491A4 EP2329491A4 (de) 2012-11-28
EP2329491B1 true EP2329491B1 (de) 2018-04-18

Family

ID=41722338

Family Applications (1)

Application Number Title Priority Date Filing Date
EP09810709.7A Active EP2329491B1 (de) 2008-08-29 2009-08-31 Hybridspracherkennung

Country Status (5)

Country Link
US (3) US7933777B2 (de)
EP (1) EP2329491B1 (de)
JP (2) JP2012501480A (de)
CA (2) CA3002206C (de)
WO (1) WO2010025440A2 (de)

Families Citing this family (88)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8335688B2 (en) * 2004-08-20 2012-12-18 Multimodal Technologies, Llc Document transcription system training
US7844464B2 (en) * 2005-07-22 2010-11-30 Multimodal Technologies, Inc. Content-based audio playback emphasis
US20120253823A1 (en) * 2004-09-10 2012-10-04 Thomas Barton Schalk Hybrid Dialog Speech Recognition for In-Vehicle Automated Interaction and In-Vehicle Interfaces Requiring Minimal Driver Processing
US7502741B2 (en) * 2005-02-23 2009-03-10 Multimodal Technologies, Inc. Audio signal de-identification
US7640158B2 (en) 2005-11-08 2009-12-29 Multimodal Technologies, Inc. Automatic detection and application of editing patterns in draft documents
US7831423B2 (en) * 2006-05-25 2010-11-09 Multimodal Technologies, Inc. Replacing text representing a concept with an alternate written form of the concept
JP5385134B2 (ja) * 2006-06-22 2014-01-08 マルチモーダル・テクノロジーズ・エルエルシー コンピュータ実装方法
US8364481B2 (en) 2008-07-02 2013-01-29 Google Inc. Speech recognition with parallel recognition tasks
US7933777B2 (en) * 2008-08-29 2011-04-26 Multimodal Technologies, Inc. Hybrid speech recognition
US8019608B2 (en) 2008-08-29 2011-09-13 Multimodal Technologies, Inc. Distributed speech recognition using one way communication
US20100125450A1 (en) 2008-10-27 2010-05-20 Spheris Inc. Synchronized transcription rules handling
US8346549B2 (en) * 2009-12-04 2013-01-01 At&T Intellectual Property I, L.P. System and method for supplemental speech recognition by identified idle resources
US20110184740A1 (en) * 2010-01-26 2011-07-28 Google Inc. Integration of Embedded and Network Speech Recognizers
JP2011232619A (ja) * 2010-04-28 2011-11-17 Ntt Docomo Inc 音声認識装置および音声認識方法
US9634855B2 (en) 2010-05-13 2017-04-25 Alexander Poltorak Electronic personal interactive device that determines topics of interest using a conversational agent
WO2011163538A1 (en) 2010-06-24 2011-12-29 Honda Motor Co., Ltd. Communication system and method between an on-vehicle voice recognition system and an off-vehicle voice recognition system
US8959102B2 (en) 2010-10-08 2015-02-17 Mmodal Ip Llc Structured searching of dynamic structured document corpuses
JP6317111B2 (ja) 2011-02-22 2018-04-25 スピーク・ウィズ・ミー・インコーポレイテッドＳｐｅａｋ Ｗｉｔｈ Ｍｅ，Ｉｎｃ． ハイブリッド型クライアントサーバ音声認識
CN103650034A (zh) * 2011-07-05 2014-03-19 三菱电机株式会社 语音识别装置及导航装置
US9009041B2 (en) * 2011-07-26 2015-04-14 Nuance Communications, Inc. Systems and methods for improving the accuracy of a transcription using auxiliary data such as personal data
JP5658641B2 (ja) * 2011-09-15 2015-01-28 株式会社Ｎｔｔドコモ 端末装置、音声認識プログラム、音声認識方法および音声認識システム
US20130085753A1 (en) * 2011-09-30 2013-04-04 Google Inc. Hybrid Client/Server Speech Recognition In A Mobile Device
US8924219B1 (en) 2011-09-30 2014-12-30 Google Inc. Multi hotword robust continuous voice command detection in mobile devices
JP5821639B2 (ja) 2012-01-05 2015-11-24 株式会社デンソー 音声認識装置
JP5957269B2 (ja) * 2012-04-09 2016-07-27 クラリオン株式会社 音声認識サーバ統合装置および音声認識サーバ統合方法
US9715879B2 (en) * 2012-07-02 2017-07-25 Salesforce.Com, Inc. Computer implemented methods and apparatus for selectively interacting with a server to build a local database for speech recognition at a device
US9583100B2 (en) * 2012-09-05 2017-02-28 GM Global Technology Operations LLC Centralized speech logger analysis
JP2014062944A (ja) * 2012-09-20 2014-04-10 Sharp Corp 情報処理装置
JP5706384B2 (ja) 2012-09-24 2015-04-22 株式会社東芝 音声認識装置、音声認識システム、音声認識方法および音声認識プログラム
KR20140058127A (ko) * 2012-11-06 2014-05-14 삼성전자주식회사 음성인식장치 및 음성인식방법
US9171066B2 (en) * 2012-11-12 2015-10-27 Nuance Communications, Inc. Distributed natural language understanding and processing using local data sources
US9704486B2 (en) * 2012-12-11 2017-07-11 Amazon Technologies, Inc. Speech recognition power management
US9190057B2 (en) * 2012-12-12 2015-11-17 Amazon Technologies, Inc. Speech model retrieval in distributed speech recognition systems
KR20140087717A (ko) * 2012-12-31 2014-07-09 삼성전자주식회사 디스플레이 장치 및 제어 방법
KR20140089863A (ko) * 2013-01-07 2014-07-16 삼성전자주식회사 디스플레이 장치, 및 이의 제어 방법, 그리고 음성 인식 시스템의 디스플레이 장치 제어 방법
KR102112742B1 (ko) * 2013-01-22 2020-05-19 삼성전자주식회사 전자장치 및 그 음성 처리 방법
JP6281202B2 (ja) * 2013-07-30 2018-02-21 株式会社デンソー 応答制御システム、およびセンター
US10885918B2 (en) 2013-09-19 2021-01-05 Microsoft Technology Licensing, Llc Speech recognition using phoneme matching
DE102013219649A1 (de) * 2013-09-27 2015-04-02 Continental Automotive Gmbh Verfahren und System zum Erstellen oder Ergänzen eines benutzerspezifischen Sprachmodells in einem mit einem Endgerät verbindbaren lokalen Datenspeicher
JP6054283B2 (ja) * 2013-11-27 2016-12-27 シャープ株式会社 音声認識端末、サーバ、サーバの制御方法、音声認識システム、音声認識端末の制御プログラム、サーバの制御プログラムおよび音声認識端末の制御方法
WO2015098079A1 (ja) * 2013-12-26 2015-07-02 パナソニックＩｐマネジメント株式会社 音声認識処理装置、音声認識処理方法、および表示装置
DE102014200570A1 (de) * 2014-01-15 2015-07-16 Bayerische Motoren Werke Aktiengesellschaft Verfahren und System zur Erzeugung eines Steuerungsbefehls
FR3016458B1 (fr) * 2014-01-16 2017-06-16 Cie Ind Et Financiere D'ingenierie Ingenico Procede de securisation d'une transaction realisee par carte bancaire
US9601108B2 (en) 2014-01-17 2017-03-21 Microsoft Technology Licensing, Llc Incorporating an exogenous large-vocabulary model into rule-based speech recognition
KR102215579B1 (ko) 2014-01-22 2021-02-15 삼성전자주식회사 대화형 시스템, 디스플레이 장치 및 그 제어 방법
US10749989B2 (en) 2014-04-01 2020-08-18 Microsoft Technology Licensing Llc Hybrid client/server architecture for parallel processing
CN104978965B (zh) 2014-04-07 2019-04-26 三星电子株式会社 电子装置及利用电子装置和服务器的语音识别执行方法
KR101585105B1 (ko) * 2014-08-11 2016-01-13 주식회사 케이티 음성 인식 장치, 방법 및 시스템
JP6118838B2 (ja) * 2014-08-21 2017-04-19 本田技研工業株式会社 情報処理装置、情報処理システム、情報処理方法、及び情報処理プログラム
US20160111090A1 (en) * 2014-10-16 2016-04-21 General Motors Llc Hybridized automatic speech recognition
JP2015143866A (ja) * 2015-02-25 2015-08-06 株式会社東芝 音声認識装置、音声認識システム、音声認識方法および音声認識プログラム
US20180211668A1 (en) * 2015-07-17 2018-07-26 Nuance Communications, Inc. Reduced latency speech recognition system using multiple recognizers
KR102365757B1 (ko) * 2015-09-09 2022-02-18 삼성전자주식회사 인식 장치, 인식 방법 및 협업 처리 장치
US9997161B2 (en) 2015-09-11 2018-06-12 Microsoft Technology Licensing, Llc Automatic speech recognition confidence classifier
US10706852B2 (en) 2015-11-13 2020-07-07 Microsoft Technology Licensing, Llc Confidence features for automated speech recognition arbitration
CN107452383B (zh) * 2016-05-31 2021-10-26 华为终端有限公司 一种信息处理方法、服务器、终端及信息处理系统
US10546061B2 (en) 2016-08-17 2020-01-28 Microsoft Technology Licensing, Llc Predicting terms by using model chunks
US11115463B2 (en) * 2016-08-17 2021-09-07 Microsoft Technology Licensing, Llc Remote and local predictions
KR101700099B1 (ko) * 2016-10-11 2017-01-31 미디어젠(주) 하이브리드 음성인식 복합 성능 자동 평가시스템
CN108010523B (zh) * 2016-11-02 2023-05-09 松下电器（美国）知识产权公司 信息处理方法以及记录介质
JP6751658B2 (ja) * 2016-11-15 2020-09-09 クラリオン株式会社 音声認識装置、音声認識システム
US10971157B2 (en) 2017-01-11 2021-04-06 Nuance Communications, Inc. Methods and apparatus for hybrid speech recognition processing
US10614804B2 (en) 2017-01-24 2020-04-07 Honeywell International Inc. Voice control of integrated room automation system
KR20180118461A (ko) * 2017-04-21 2018-10-31 엘지전자 주식회사 음성 인식 장치 및 음성 인식 방법
US10984329B2 (en) 2017-06-14 2021-04-20 Ademco Inc. Voice activated virtual assistant with a fused response
JP6570796B2 (ja) * 2017-06-22 2019-09-04 三菱電機株式会社 音声認識装置および音声認識方法
US10515637B1 (en) 2017-09-19 2019-12-24 Amazon Technologies, Inc. Dynamic speech processing
KR102471493B1 (ko) * 2017-10-17 2022-11-29 삼성전자주식회사 전자 장치 및 음성 인식 방법
US11597519B2 (en) 2017-10-17 2023-03-07 The Boeing Company Artificially intelligent flight crew systems and methods
DE102017222549A1 (de) 2017-12-13 2019-06-13 Robert Bosch Gmbh Steuerverfahren und Sprachdialogsystem
US10192554B1 (en) * 2018-02-26 2019-01-29 Sorenson Ip Holdings, Llc Transcription of communications using multiple speech recognition systems
KR102517228B1 (ko) * 2018-03-14 2023-04-04 삼성전자주식회사 사용자의 입력에 대한 외부 전자 장치의 응답 시간에 기반하여 지정된 기능을 제어하는 전자 장치 및 그의 방법
US20190332848A1 (en) 2018-04-27 2019-10-31 Honeywell International Inc. Facial enrollment and recognition system
US10147428B1 (en) * 2018-05-30 2018-12-04 Green Key Technologies Llc Computer systems exhibiting improved computer speed and transcription accuracy of automatic speech transcription (AST) based on a multiple speech-to-text engines and methods of use thereof
US20190390866A1 (en) 2018-06-22 2019-12-26 Honeywell International Inc. Building management system with natural language interface
KR102611386B1 (ko) * 2018-06-27 2023-12-08 구글 엘엘씨 로컬 텍스트-응답 맵을 활용하여 사용자의 음성 발화에 대한 응답 렌더링
US11094326B2 (en) * 2018-08-06 2021-08-17 Cisco Technology, Inc. Ensemble modeling of automatic speech recognition output
US11967318B2 (en) * 2019-01-08 2024-04-23 Samsung Electronics Co., Ltd. Method and system for performing speech recognition in an electronic device
WO2020245912A1 (ja) * 2019-06-04 2020-12-10 日本電信電話株式会社 音声認識制御装置、音声認識制御方法、およびプログラム
KR102266062B1 (ko) * 2019-07-01 2021-06-17 주식회사 한글과컴퓨터 분할 음성 인식을 기반으로 발화자의 음성에 대응하는 텍스트의 생성이 가능한 음성 인식 장치 및 그 동작 방법
WO2021029642A1 (en) 2019-08-13 2021-02-18 Samsung Electronics Co., Ltd. System and method for recognizing user's speech
WO2021029627A1 (en) 2019-08-13 2021-02-18 Samsung Electronics Co., Ltd. Server that supports speech recognition of device, and operation method of the server
WO2021029643A1 (en) 2019-08-13 2021-02-18 Samsung Electronics Co., Ltd. System and method for modifying speech recognition result
KR20210042520A (ko) * 2019-10-10 2021-04-20 삼성전자주식회사 전자 장치 및 이의 제어 방법
JP2020129130A (ja) * 2020-04-27 2020-08-27 パイオニア株式会社 情報処理装置
CN111627431B (zh) * 2020-05-13 2022-08-09 广州国音智能科技有限公司 语音识别方法、装置、终端及存储介质
CN111681647B (zh) * 2020-06-10 2023-09-05 北京百度网讯科技有限公司 用于识别词槽的方法、装置、设备以及存储介质
CN112164392A (zh) * 2020-11-13 2021-01-01 北京百度网讯科技有限公司 确定显示的识别文本的方法、装置、设备以及存储介质

Family Cites Families (34)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5101375A (en) * 1989-03-31 1992-03-31 Kurzweil Applied Intelligence, Inc. Method and apparatus for providing binding and capitalization in structured report generation
US5365574A (en) * 1990-05-15 1994-11-15 Vcs Industries, Inc. Telephone network voice recognition and verification using selectively-adjustable signal thresholds
ZA948426B (en) * 1993-12-22 1995-06-30 Qualcomm Inc Distributed voice recognition system
US6665639B2 (en) * 1996-12-06 2003-12-16 Sensory, Inc. Speech recognition in consumer electronic products
US6101473A (en) * 1997-08-08 2000-08-08 Board Of Trustees, Leland Stanford Jr., University Using speech recognition to access the internet, including access via a telephone
US6125345A (en) * 1997-09-19 2000-09-26 At&T Corporation Method and apparatus for discriminative utterance verification using multiple confidence measures
US6006183A (en) * 1997-12-16 1999-12-21 International Business Machines Corp. Speech recognition confidence level display
JP4243019B2 (ja) * 1998-04-03 2009-03-25 ヴァーティカル・ネットワークス・インコーポレーテッド インテリジェント・ブリッジｔｄｍおよびパケット・バスを用いて音声およびデータを伝送するシステムおよび方法
US6154465A (en) * 1998-10-06 2000-11-28 Vertical Networks, Inc. Systems and methods for multiple mode voice and data communications using intelligenty bridged TDM and packet buses and methods for performing telephony and data functions using the same
US6229880B1 (en) * 1998-05-21 2001-05-08 Bell Atlantic Network Services, Inc. Methods and apparatus for efficiently providing a communication system with speech recognition capabilities
US7003463B1 (en) * 1998-10-02 2006-02-21 International Business Machines Corporation System and method for providing network coordinated conversational services
US6377922B2 (en) * 1998-12-29 2002-04-23 At&T Corp. Distributed recognition system having multiple prompt-specific and response-specific speech recognizers
EP1088299A2 (de) 1999-03-26 2001-04-04 Scansoft, Inc. Client-server-spracherkennungssystem
US6363349B1 (en) 1999-05-28 2002-03-26 Motorola, Inc. Method and apparatus for performing distributed speech processing in a communication system
US6292781B1 (en) * 1999-05-28 2001-09-18 Motorola Method and apparatus for facilitating distributed speech processing in a communication system
US7203651B2 (en) * 2000-12-07 2007-04-10 Art-Advanced Recognition Technologies, Ltd. Voice control system with multiple voice recognition engines
US6785654B2 (en) 2001-11-30 2004-08-31 Dictaphone Corporation Distributed speech recognition system with speech recognition engines offering multiple functionalities
GB2383459B (en) * 2001-12-20 2005-05-18 Hewlett Packard Co Speech recognition system and method
US6898567B2 (en) * 2001-12-29 2005-05-24 Motorola, Inc. Method and apparatus for multi-level distributed speech recognition
JP2004012653A (ja) * 2002-06-05 2004-01-15 Matsushita Electric Ind Co Ltd 音声認識システム、音声認識クライアント、音声認識サーバ、音声認識クライアントプログラムおよび音声認識サーバプログラム
JP3759508B2 (ja) * 2003-03-31 2006-03-29 オリンパス株式会社 アクチュエータ及びアクチュエータ駆動方法並びにアクチュエータシステム
US20040210443A1 (en) * 2003-04-17 2004-10-21 Roland Kuhn Interactive mechanism for retrieving information from audio and multimedia files containing speech
US7363228B2 (en) * 2003-09-18 2008-04-22 Interactive Intelligence, Inc. Speech recognition system and method
JP2005249829A (ja) 2004-03-01 2005-09-15 Advanced Media Inc 音声認識を行うコンピュータネットワークシステム
US20050215260A1 (en) 2004-03-23 2005-09-29 Motorola, Inc. Method and system for arbitrating between a local engine and a network-based engine in a mobile communication network
JP4554285B2 (ja) * 2004-06-18 2010-09-29 トヨタ自動車株式会社 音声認識システム、音声認識方法、及び音声認識プログラム
US8589156B2 (en) * 2004-07-12 2013-11-19 Hewlett-Packard Development Company, L.P. Allocation of speech recognition tasks and combination of results thereof
US7437297B2 (en) * 2005-01-27 2008-10-14 International Business Machines Corporation Systems and methods for predicting consequences of misinterpretation of user commands in automated systems
KR101073190B1 (ko) 2005-02-03 2011-10-13 주식회사 현대오토넷 분산 음성 인식 시스템을 이용한 텔레매틱스 시스템의정보 제공 시스템 및 방법
JP2007033901A (ja) 2005-07-27 2007-02-08 Nec Corp 音声認識システム、音声認識方法、および音声認識用プログラム
US8612230B2 (en) * 2007-01-03 2013-12-17 Nuance Communications, Inc. Automatic speech recognition with a selection list
US8019608B2 (en) * 2008-08-29 2011-09-13 Multimodal Technologies, Inc. Distributed speech recognition using one way communication
US7933777B2 (en) * 2008-08-29 2011-04-26 Multimodal Technologies, Inc. Hybrid speech recognition
US8150696B2 (en) * 2008-12-08 2012-04-03 At&T Intellectual Property I, L.P. Method of providing dynamic speech processing services during variable network connectivity

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
None *

Also Published As

Publication number Publication date
EP2329491A2 (de) 2011-06-08
JP2013232001A (ja) 2013-11-14
CA3002206A1 (en) 2010-03-04
WO2010025440A2 (en) 2010-03-04
CA2732255C (en) 2019-02-05
US20110238415A1 (en) 2011-09-29
JP2012501480A (ja) 2012-01-19
CA2732255A1 (en) 2010-03-04
EP2329491A4 (de) 2012-11-28
JP6113008B2 (ja) 2017-04-12
US7933777B2 (en) 2011-04-26
US20100057450A1 (en) 2010-03-04
US8249877B2 (en) 2012-08-21
CA3002206C (en) 2020-04-07
US20120296644A1 (en) 2012-11-22
WO2010025440A3 (en) 2010-06-03

Similar Documents

Publication Publication Date Title
EP2329491B1 (de) Hybridspracherkennung
JP7359886B2 (ja) 強化された発話エンドポイント指定のための方法、システム、およびコンピュータ可読記憶媒体
US10339917B2 (en) Enhanced speech endpointing
EP3163568B1 (de) Endpunktbestimmung von sprache
US9196250B2 (en) Application services interface to ASR
US20060229873A1 (en) Methods and apparatus for adapting output speech in accordance with context of communication
US11763819B1 (en) Audio encryption
US20230025709A1 (en) Transferring dialog data from an initially invoked automated assistant to a subsequently invoked automated assistant
AU2023274128A1 (en) Hot-word free pre-emption of automated assistant response presentation
EP2733697A1 (de) Anwendungsdienst-Schnittstelle zu ASR
US12125477B2 (en) Hot-word free pre-emption of automated assistant response presentation
KR20240089626A (ko) 음성 발언 완료를 위한 연속 컨텐츠의 선택적 생성 및/또는 선택적 렌더링

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20110329

AK Designated contracting states

Kind code of ref document: A2

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

AX Request for extension of the european patent

Extension state: AL BA RS

DAX Request for extension of the european patent (deleted)
RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: MULTIMODAL TECHNOLOGIES, LLC

A4 Supplementary search report drawn up and despatched

Effective date: 20121025

RIC1 Information provided on ipc code assigned before grant

Ipc: G10L 15/28 20060101AFI20121019BHEP

17Q First examination report despatched

Effective date: 20170529

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Ref document number: 602009051892

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: G10L0019000000

Ipc: G10L0015320000

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

RIC1 Information provided on ipc code assigned before grant

Ipc: G10L 15/30 20130101ALI20171220BHEP

Ipc: G10L 15/32 20130101AFI20171220BHEP

INTG Intention to grant announced

Effective date: 20180122

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 991271

Country of ref document: AT

Kind code of ref document: T

Effective date: 20180515

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602009051892

Country of ref document: DE

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 10

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20180418

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180718

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180718

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180719

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 991271

Country of ref document: AT

Kind code of ref document: T

Effective date: 20180418

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180820

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602009051892

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

REG Reference to a national code

Ref country code: CH

Ref legal event code: PK

Free format text: BERICHTIGUNGEN

RIC2 Information provided on ipc code assigned after grant

Ipc: G10L 15/32 20130101AFI20171220BHEP

Ipc: G10L 15/30 20130101ALI20171220BHEP

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

26N No opposition filed

Effective date: 20190121

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180831

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180831

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180831

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20180831

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180831

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180831

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20090831

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180831

Ref country code: MK

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180418

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180818

REG Reference to a national code

Ref country code: GB

Ref legal event code: 732E

Free format text: REGISTERED BETWEEN 20220120 AND 20220126

REG Reference to a national code

Ref country code: DE

Ref legal event code: R081

Ref document number: 602009051892

Country of ref document: DE

Owner name: 3M INNOVATIVE PROPERTIES CO., ST. PAUL, US

Free format text: FORMER OWNER: MULTIMODAL TECHNOLOGIES, LLC, PITTSBURGH, PA., US

Ref country code: DE

Ref legal event code: R081

Ref document number: 602009051892

Country of ref document: DE

Owner name: SOLVENTUM INTELLECTUAL PROPERTIES CO. (N.D.GES, US

Free format text: FORMER OWNER: MULTIMODAL TECHNOLOGIES, LLC, PITTSBURGH, PA., US

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20220721

Year of fee payment: 14

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230530

REG Reference to a national code

Ref country code: DE

Ref legal event code: R081

Ref document number: 602009051892

Country of ref document: DE

Owner name: SOLVENTUM INTELLECTUAL PROPERTIES CO. (N.D.GES, US

Free format text: FORMER OWNER: 3M INNOVATIVE PROPERTIES CO., ST. PAUL, MN, US

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230831

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20240723

Year of fee payment: 16

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20240723

Year of fee payment: 16