RU2778380C2 - Method and system for speech conversion into text - Google Patents

Abstract

FIELD: speech recognition.

SUBSTANCE: group of inventions relates to the field of speech recognition; it can be used for speech conversion into text. The method contains stages, at which a user statement corresponding to speech is received, the first prediction text corresponding to the user statement and the first reliability indicator corresponding to the first prediction text are determined, using a local graph, the user statement is transmitted to a server, the second prediction text corresponding to the user statement and the second reliability indicator corresponding to the second prediction text are received from the server, if the first reliability indicator is higher than the second reliability indicator, the first prediction text is output.

EFFECT: increase in the accuracy of recognition.

17 cl, 8 dwg

Description

[001] The present technology relates to speech recognition in general, and more specifically, to a method and system for translating speech into text.

BACKGROUND OF THE INVENTION

[002] Speech recognition can be performed by the device to make user interaction with the device easier and more natural. Instead of entering text or entering commands through the touch screen, the user can control the device with their speech. Various methods have been developed to translate user speech into text. A model that includes a dictionary can be used to identify the text that the user has spoken.

[003] The model can be developed using a corpus of text, such as text collected from dictionaries and other books, newspapers and magazines, and so on. The model may indicate the probability of occurrence of some particular word in speech. The model can be considered a "universal" model and can be used for numerous devices. For example, a single model could be developed for all US English speakers. Although these generic models can be used for a large number of users, they are not personalized for each user.

[004] US Pat. No. 9,147,395, issued September 29, 2015 to LG Electronics Inc., describes a mobile terminal and a voice recognition method used therein. The speech recognition method may include receiving the user's speech; providing the received speech to a first speech recognizer provided in the server and a second speech recognizer provided in the mobile terminal; obtaining first speech recognition data as a result of recognizing the received speech by the first speech recognizer; obtaining second speech recognition data as a result of recognizing the received speech by the second speech recognizer; evaluating a function corresponding to the user's intent based on at least one of the first and second speech recognition data; calculating similarity between the first and second speech recognition data when the evaluated function requires personal information; and selecting one of the first and second speech recognition data based on the calculated similarity.

[005] Patent Application Publication US 2014/0136210 A1, filed Nov. 14, 2012 to Nuance Communications Inc., describes that personalization of speech recognition while maintaining the privacy of user data is achieved by passing data associated with received speech to a speech recognition service. and receiving a result from the speech recognition service. The result of the speech recognition service is generated based on the general language model of speech. The system generates an input state machine from the speech recognition result, and generates an input state machine with the phone edit state machine to obtain the resulting state machine. The system generates the resulting state machine with the user data state machine to obtain the second resulting state machine, and uses the best path through the second the resulting state machine for obtaining a user-specific speech recognition result.

[006] Patent Application Publication US 2015/0081294 A1, filed September 18, 2014 by Maluuba Inc., describes a system, method, and computer program for generating customizable text representations of audio commands. The first speech recognition module may be used to generate a first textual representation of the audio command based on a common language grammar. A second speech recognition module may be used to generate a second textual representation of the audio command, the second module including a custom language grammar that may include contacts for some particular user. Object extraction is applied to the second text representation and the objects are checked against the file containing the personal language style. If objects are found in the user's native language, the two text representations can be combined into a combined text representation, and named object recognition can be performed again to retrieve additional objects.

SUMMARY OF THE INVENTION

[007] The developers of the present technology have recognized at least one technical problem associated with prior art approaches.

[008] The present technology relates to speech recognition and, more specifically, to methods and systems for translating speech into text. Users of client devices may include personal data in their speech. Some of this personal data may be stored on their client devices. For example, the user may request to call a contact stored on his client device or play a song stored on his client device. The generic model may not work correctly when interpreting such queries. The universal model may not reflect that a user is more likely than the general population to pronounce the words in their personal data. The user experience can be improved if the system for translating speech into text is adapted to the user's personal data.

[009] As mentioned above, speech recognition models are usually based on a universal text corpus and are applied to a large number of users. A model tailored to a specific person may provide more accurate results.

[0010] A speech-to-text translation system that uses a generic model may be supplemented by a second model, which is referred to herein as a local graph. The local graph may be specific to an individual user and may be generated using text stored on that user's client device. The local graph may be stored on the client device.

[0011] After receiving a user utterance, for example, via a microphone in a client device, the user utterance may be entered into a local graph and a universal graph. Both the local graph and the universal graph can output predictive text for the user's statement and/or a confidence score corresponding to the prediction. The confidence scores can be compared and either the predictive text from the local graph or the predictive text from the generic graph can be selected.

[0012] According to a first broad aspect of the present technology, a method is provided for translating speech into text, wherein the speech is received by a client device. The method is executable on a client device and comprises: receiving a user statement corresponding to speech; determining, based on the local graph stored by the client device, a first predictive text corresponding to the user utterance and a first confidence score corresponding to the first predictive text, wherein the local graph contains a tree, each leaf of the tree corresponding to a word, and wherein the local graph is generated based on text stored by the client device; transferring a user statement to the server; receiving from the server a second predictive text corresponding to the user statement and a second confidence indicator corresponding to the second predictive text; comparing the first confidence score with the second confidence score; and after determining that the first confidence score is greater than the second confidence score, outputting the first predictive text.

[0013] In some implementations of the above method, the first node in each branch of the tree contains the first letter of the word.

[0014] In some implementations of the above method, each subsequent node after the first node contains the next letter of the word.

[0015] In some implementations of the above method, the first leaf in the tree contains the first conjugation of the word, while the second leaf in the tree contains the second conjugation of the said word, and the first leaf and the second leaf have a common parent node in the tree.

[0016] In some implementations of the above method, each leaf of the tree is associated with a confidence score, wherein the confidence score indicates the frequency with which the word corresponding to the corresponding leaf occurs in the text stored by the client device.

[0017] In some implementations of the above method, the definition of the first predictive text contains the definition of a leaf of the tree that corresponds to the user statement, while the first confidence indicator contains a confidence score corresponding to this leaf.

[0018] In some implementations of said method, a local graph is generated by: applying text stored by the client device to a normalizer (“normalizer”), thereby generating normalized text; applying the text stored by the client device and the normalized text to the inflector (“inflector”), thereby generating the inflectored text; and generating a local graph based on the text stored by the client device, the normalized text, and the inflected text.

[0019] In some implementations of the mentioned method, the normalizer converts the digits in the text stored by the client device into text.

[0020] In some implementations of the above method, the inflector takes a word and generates multiple forms of that word.

[0021] In some implementations of the above method, the text stored by the client device contains a list of contacts.

[0022] In some implementations of the above method, the text stored by the client device corresponds to the music stored by the client device.

[0023] In some implementations of the above method, the local graph is personalized for the user of the client device.

[0024] In some implementations of the above method, the local graph is a tree of n-grams, with each node in the local graph corresponding to a separate n-gram.

[0025] According to another broad aspect of the present technology, a method is provided for translating speech into text, wherein the speech is received by a client device. The method is executable on the server and comprises: receiving from the client device a user statement corresponding to speech; determining, based on the graph, a first predictive text corresponding to the user utterance and a first confidence score corresponding to the first predictive text, the graph comprising a tree, each leaf of the tree corresponding to a word; receiving from the client device a second prediction text corresponding to the user utterance and a second confidence indicator corresponding to the second prediction text, wherein the second prediction text was determined by the client device, wherein the second prediction text was determined based on a local graph stored by the client device, wherein the local graph was generated based on the text stored by the client device; comparing the first confidence score with the second confidence score; and after determining that the first confidence score is greater than the second confidence score, transmitting the first predictive text to the client device.

[0026] In some implementations of the above method, the first leaf in the tree contains the first conjugation of the word, and the second leaf in the tree contains the second conjugation of the said word, and the first leaf and the second leaf have a common parent node in the tree.

[0027] In some implementations of the method, each leaf of the tree is associated with a confidence score, where the confidence score indicates the frequency with which the word corresponding to the corresponding leaf occurs in the text stored by the client device.

[0028] According to another broad aspect of the present technology, a method is provided for translating speech into text, wherein the speech is received by a client device. The method is executable on the server and comprises: receiving from the client device a user statement corresponding to speech; determining, based on the graph, a first predictive text corresponding to the user utterance and a first confidence score corresponding to the first predictive text, the graph comprising a tree, each leaf of the tree corresponding to a word; and transmitting to the client device the first prediction text and the first confidence score for comparing the first prediction text and the first confidence score with the second prediction text and the second confidence score, wherein the second prediction text and the second confidence score have been determined by the client device based on a local graph stored by the client device. , while the local graph was generated based on the text stored by the client device.

[0029] In some implementations of the above method, each leaf of the tree is associated with a confidence score.

[0030] In some implementations of said method, the graph corresponds to a graph used to translate speech to text for multiple client devices, with the local graph being unique to the client device.

[0031] In some implementations of said method, a local graph was generated based on the text stored by the client device, the normalized text, and the inflected text, wherein the normalized text was generated by applying the text stored by the client device to the normalizer, and the inflected text was generated by applying the text stored by the client device and the normalized text to the inflector.

[0032] According to another broad aspect of the present technology, a method is provided for translating speech into text, wherein the speech is received by a client device. The method is executable on a client device and comprises: receiving a user statement corresponding to speech; determining, based on the local graph stored by the client device, the predictive text corresponding to the user statement and the confidence score corresponding to the predictive text, where the local graph contains a tree, where each leaf of this tree corresponds to a word, and where the local graph is generated based on the text , stored by the client device; comparing a confidence score with a threshold confidence score; determining, based on the confidence threshold, whether the user statement should be transmitted to the server; after determining that the confidence score satisfies the threshold confidence score, determining that the user statement should not be transmitted to the server; and output of predictive text.

[0033] As used herein, a "server" is a system and/or computer program that runs on appropriate hardware and is capable of receiving requests (eg, from electronic devices) over a network and fulfilling or causing those requests to be executed. The hardware may be one physical computer or one physical computer system, but neither is required for the present technology. In the present context, the use of the term "server" is not intended to imply that every task (e.g., instructions or requests received) or any particular task will be accepted, executed, or called for execution by the same server (i.e., the same software and/or hardware); it is understood that any number of software elements or hardware devices may be involved in receiving/sending, executing or invoking to perform any task or request, or the consequences of any task or request; and all of this software and hardware may be a single server or multiple servers, both of which are included in the expression "at least one server".

[0034] As used herein, an "electronic device" is any computer equipment that is capable of executing software that is appropriate for its respective task. Thus, some (non-limiting) examples of electronic devices include personal computers (desktops, laptops, netbooks, etc.), smartphones and tablets, and network equipment such as routers, switches, and gateways. It should be noted that a device acting as an electronic device in the present context is not excluded from acting as a server for other electronic devices. The use of the term "electronic device" does not preclude the use of multiple electronic devices in receiving/sending, performing or invoking to perform any task or request, or the consequences of any task or request, or the steps of any method described herein.

[0035] As used herein, a "database" is any structured collection of data, regardless of its specific structure, database administration software, or computer hardware on which data is stored, implemented, or otherwise made available for use. . The database may reside on the same hardware as the process that stores or uses the information stored in the database, or it may reside on separate hardware such as a dedicated server or multiple servers.

[0036] In the context of the present description, the expression "information" includes information of any nature or form that is capable of being stored in the database in any way. Thus, information includes, but is not limited to, audiovisual works (images, films, sound recordings, presentations, etc.), data (location data, numerical data, etc.), text (opinions, comments , questions, messages, etc.), documents, spreadsheets, etc.

[0037] In the context of the present description, the expression "media used by the computer" is intended to include media of any type and form, including RAM, ROM, disks (CD-ROMs, DVDs, floppy disks, hard disk drives, etc.), etc.), USB keys, SSDs, tape drives, etc.

[0038] As used herein, unless expressly provided otherwise, an "indication" of an information element may be the information element itself or a pointer, link, hyperlink, or other indirect mechanism that allows the recipient of such indication to locate a location in a network, memory, database, or other a computer-readable medium from which the information element can be retrieved. For example, a document designation may include the document itself (i.e., its contents), or it may be a unique document descriptor identifying a file with respect to some particular file system, or some other means of directing the recipient of such designation to a location on the network, an address in memory, a table in a database, or some other location where the file can be accessed. One skilled in the art will appreciate that the degree of accuracy required in such an indication depends on the degree of any prior understanding of what interpretation will be given of the information exchanged between the sender and recipient of that indication. For example, if prior to communication between sender and receiver it is understood that the indication of the information element will be in the form of a database key to be entered in some particular table of the predefined database containing the information element, then sending the database key is all that is required to efficiently transmit the information element. recipient, even if the information element itself has not been transmitted between the sender and recipient of the indication.

[0039] In the context of the present description, the words "first", "second", "third", etc. are used as adjectives only to allow the nouns they modify to be distinguished from each other, and not to describe any particular relationship between such nouns. Thus, for example, it should be understood that the use of the terms "first server" and "third server" does not imply any particular order, type, chronology, hierarchy or ranking (for example) of such servers, nor does their use (in itself ) does not mean that some "second server" must necessarily exist in any given situation. Also, as discussed elsewhere in this document, reference to a "first" element and a "second" element does not exclude that the two elements are in fact the same real world element. Thus, for example, in some cases, the "first" server and the "second" server may be the same software and/or hardware, and/or they may share software and/or hardware. In other cases, they may be different software and/or hardware.

[0040] Each of the implementations of the present technology has at least one of the above aspects and/or goals, but does not necessarily have them all. It should be understood that some aspects of the present technology that have arisen in an attempt to achieve the aforementioned goal may not satisfy this goal and/or satisfy other goals that are not explicitly described in this document.

[0041] Additional and/or alternative features, aspects, and advantages of implementations of the present technology will become apparent from the following description, the accompanying drawings, and the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[0042] For a better understanding of the present technology, as well as other aspects and additional features thereof, reference is made to the following description, which is to be used in conjunction with the accompanying drawings, in which:

[0043] Figure 1 is an illustration of the components and features of a computing device in accordance with non-limiting embodiments of the present technology.

[0044] Figure 2 illustrates a diagram of a speech-to-text translation system implemented in accordance with non-limiting embodiments of the present technology.

[0045] Figure 3 illustrates a diagram of a user utterance processing system in accordance with non-limiting embodiments of the present technology.

[0046] Figure 4 illustrates a graph in accordance with non-limiting embodiments of the present technology.

[0047] Figure 5 illustrates a diagram of a local graph generation system in accordance with non-limiting embodiments of the present technology.

[0048] Figures 6, 7, and 8 illustrate a flowchart of a method for translating speech into text in accordance with non-limiting embodiments of the present technology.

DETAILED DESCRIPTION

[0049] The examples and conventions provided herein are intended primarily to help the reader understand the principles of the present technology, and not to limit its scope to such specific examples and terms. It should be understood that those skilled in the art will be able to develop various mechanisms that, although not explicitly described in this document, nevertheless embody the principles of the present technology and are included in its essence and scope.

[0050] In addition, the following description may describe implementations of the present technology in a relatively simplified manner for purposes of ease of understanding. Those skilled in the art will appreciate that various implementations of the present technology may be more complex.

[0051] In some cases, examples of modifications to the present technology that are considered useful may also be set forth. This is only to aid understanding and, again, not to rigorously define the scope or boundaries of the present technology. These modifications are not an exhaustive list, and other modifications may be made by one skilled in the art while still remaining within the scope of the present technology. In addition, cases where examples of modifications are not given should not be interpreted to mean that no modifications can be made and/or that what is described is the only way to implement such an element of the present technology.

[0052] In addition, all statements contained herein that indicate principles, aspects, and implementations of the present technology, as well as specific examples thereof, are intended to cover both structural and functional equivalents, whether or not they are currently known. time or will be developed in the future. Thus, for example, those skilled in the art will appreciate that any block diagrams herein represent conceptual views of an illustrative circuit embodying the principles of the present technology. Likewise, it will be understood that any and all flowcharts, flow charts, state transition diagrams, pseudocodes, and the like represent various processes that may per se be represented on computer-readable media and executed by a computer or processor, regardless of whether such a computer is depicted. or processor explicitly or not.

[0053] The functions of the various elements shown in the figures, including any functional block labeled "processor" or "graphics processing unit", can be provided using special hardware, as well as hardware capable of executing software, in combination with the right software. When provided by a processor, the functions may be provided by one dedicated processor, one shared processor, and/or multiple separate processors, some of which may be shared. In some embodiments of the present technology, the processor may be a general purpose processor such as a central processing unit (CPU) or a processor dedicated to a specific purpose such as a graphics processing unit (GPU). In addition, explicit use of the term "processor" or "controller" should not be construed as referring solely to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (DSP) hardware, network a processor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), read-only memory (ROM) for storing software, random access memory (RAM), and/or non-volatile storage. Other hardware, traditional and/or specialized, may also be included.

[0054] Software modules, or simply modules, which may be understood as software, may be represented herein as any combination of flowchart elements or other elements indicating the execution of process steps and/or a textual description. Such modules may be implemented in hardware that is shown explicitly or implicitly.

[0055] With these fundamentals in mind, consider some non-limiting examples to illustrate various implementations of aspects of the present technology.

[0056] With reference to Figure 1, a computing device 100 is shown suitable for use with some implementations of the present technology. Computing device 100 includes various hardware components, including one or more single or multi-core processors collectively represented by processor 110, graphics processing unit (GPU) 111, solid state drive 120, RAM 130, display interface 140, and input/output interface 150.

[0057] Communication between various components of computing device 100 may be provided by one or more internal and/or external buses 160 (eg, PCI bus, Universal Serial Bus, IEEE 1394 "Firewire" bus, SCSI bus, Serial-ATA bus, etc. .), with which various hardware components are electronically linked.

[0058] I/O interface 150 may be coupled to touch screen 190 and/or one or more internal and/or external buses 160. Touch screen 190 may be part of a display. In some embodiments, the touch screen 190 is a display. Touch screen 190 may also be referred to as screen 190. In the embodiments illustrated in Figure 1, touch screen 190 includes touch sensing hardware 194 (e.g., pressure-sensitive cells embedded in the display layer to allow detection of physical interaction between the user and the display ) and a touch input/output controller 192 capable of communicating with the display interface 140 and/or one or more internal and/or external buses 160. In some embodiments, the input/output interface 150 may be connected to a keyboard (not shown), a mouse (not shown) or a trackpad (not shown) allowing a user to interact with computing device 100 in addition to or instead of touch screen 190. In some embodiments, computing device 100 may include one or more microphones (not shown). Microphones can record audio, such as user utterances. User statements may be translated into commands for controlling computing device 100.

[0059] It should be noted that certain components of computing device 100 may be omitted from certain non-limiting embodiments of the present technology. For example, touch screen 190 may be omitted, especially (but not limited to) when the computing device is implemented as a smart speaker.

[0060] According to implementations of the present technology, the solid state drive 120 stores program instructions suitable for loading into main memory 130 and execution by processor 110 and/or GPU 111. For example, the program instructions may be part of a library or application.

[0061] Computing device 100 may be a server, desktop computer, laptop, tablet, smartphone, personal digital assistant, or any device that can be configured to implement the present technology, as will be appreciated by one of skill in the art.

[0062] With reference to Figure 2, a diagram of a speech-to-text translation system implemented in accordance with non-limiting embodiments of the present technology is illustrated. Two client devices 210 and 220 are illustrated. Client devices 210 and 220 may be computing devices 100 such as smartphones, tablets, or smart speakers. Each of the client devices 210 and 220 can be controlled by different users.

[0063] Client devices 210 and 220 may store various data corresponding to the user of each device. For example, client devices 210 and 220 may store contacts, emails, text messages, music, web browsing history, calendar data, applications, and so on. While described as being stored on client devices 210 and 220, some or all of this data may be stored remotely from client devices 210 and 220. For example, the contact list used by client device 210 may be stored in a cloud storage environment or on a remote server.

[0064] Data stored on client device 210 may be used to generate a local graph 215. Local graph 215 may be generated by client device 210. Local graph 215 may be stored on client device 210. Local graph 215 may include a dictionary of words present in the data stored on the client device 210. The local graph 215 may include a predictive probability that each word in the graph will be included in the text. The predictive probability may be determined based on the frequency of a word in the data stored on the client device 210. 225 may be generated for client device 220 based on data stored on client device 220.

[0065] Local graphs 215 and 225 can be used to predict words that occur in a user statement. In addition to the prediction, local graphs 215 and 225 can be used to determine a confidence score corresponding to the prediction. Since local graphs 215 and 225 are generated using data stored on client devices 210 and 220, each of these local graphs 215 and 225 is personalized for an associated client device 210 or 220. If both local graphs 215 and 225 received identical input data, local columns 215 and 225 may give different predictions.

[0066] The client devices 210 and 220 may communicate with the server 230, such as through a network connection. Server 230 may provide an interface for translating speech into text. Server 230 may store graph 235. Graph 235 may have the same or similar format as local graphs 215 and 225. But unlike local graphs 215 and 225, graph 235 can be used to handle requests received from multiple client devices, such as hundreds of thousands of client devices. Client devices 210 and 220 may send user utterances to server 230. Server 230 may then use graph 235 to predict text corresponding to user utterances. Graph 235 may output a confidence score for each prediction.

[0067] The confidence scores output by local graphs 215 and 225 can be compared with graph 235. If local graph 215 or 225 has a higher confidence score, the prediction made by local graph 215 or 225 can be used. If, on the other hand, graph 235 has a higher confidence score for some prediction, this prediction output by column 235 can be used.

[0068] Predictions generated using both local graph 215 or 225 and graph 235 can be more accurate than predictions generated using only local graph 215 or 225 and more accurate than predictions generated using only graph 235. By using both a local graph 215 or 225 and a graph 235, the predictions have the advantage of being tailored to the individual user due to the local graph 215 or 225 and also benefit from a larger generic data set due to the graph 235.

[0069] With reference to Figure 3, a diagram of a system for processing a user utterance is illustrated in accordance with non-limiting embodiments of the present technology. The user utterance 310 may be recorded speech, such as a spoken command from a user of the client device 210. The user utterance 310 may be recorded after the trigger word to start recording is found. For example, if the trigger word is "Alice", a custom utterance can be recorded after the word "Alice".

[0070] The user utterance may be recorded by one or more microphones, such as one or more microphones, in the client device 210. The user utterance may be recorded by a microphone external to the client device 210, such as the microphone of a headset wirelessly communicating with the client device 210.

[0071] Upon receiving the user utterance 310, the client device 210 uses the local graph 215 to determine a local prediction 320 for the user utterance 310. A confidence score 325 may be determined for the local prediction 320. The local prediction 320 may be a predictive (predicted) text corresponding to the user statement 310. Confidence metric 325 may indicate confidence associated with local prediction 320.

[0072] The client device 210 may transmit the user utterance 310 to the server 230. It should be noted that the client device 210 may transmit the user utterance to the server 230 as soon as the client device 210 receives the user utterance 310, so that the local processing described above and the server the processing to be described below are executed substantially in parallel. Other implementations are also possible, as will be described in more detail below. Server 230 may use graph 235 to determine prediction 330 for user utterance 310. Server 230 may use graph 235 to determine a confidence score 335 for prediction 330. Similar to local prediction 320, prediction 330 may include predictive text corresponding to user utterance 310. Score 335 confidence may indicate the level of confidence in the prediction 330.

[0073] Comparison engine 340 can receive and compare confidence score 325 and confidence score 335. Comparator 340 may receive predictions 320 and 330 and/or confidence metrics 325 and 335. Comparator 340 may then determine an output prediction 350. Comparator 340 may be located on client device 210, server 230, and/or another device. Comparator 340 can select which prediction 320 or 330 has the highest confidence score 325 or 335 . The selected prediction 320 or 330 may be output as output prediction 350.

[0074] In some cases, a threshold value may be used at which the local prediction 320 can be selected even if the confidence score 325 is lower than the confidence score 335. In this case, if the confidence score 325 is lower than the confidence score 335 by less than a predetermined threshold, then the local prediction 320 may be output as the output prediction 350 by the comparison engine 340 .

[0075] User utterance 310 may not be sent to server 230 for prediction using graph 235. If confidence score 325 satisfies a predetermined threshold, user utterance 310 may not be sent to server 230. In this case, output prediction 350 will be local prediction 320. Because prediction 330 will not be generated in this case, comparator 340 may not be used.

[0076] Referring to Figure 4, illustrated is a graph 400 according to non-limiting embodiments of the present technology. Graph 400 includes nodes and edges connecting those nodes. The nodes are illustrated as containing single letters. Although they are illustrated as including letters, in some cases each node may contain multiple letters, such as n-grams (such as bigrams, trigrams, or the like). Boxes 215, 225, and 235 may have the same or similar format as box 400 illustrated.

[0077] The format of graph 400 is a prefix tree. In this tree, each node is connected by exactly one edge. At the first level of this tree, there are 26 roots corresponding to the letters "A" - "Z", as an example, if the English alphabet is used. It should be noted that this tree may have more roots, for example, 33 roots, if the Russian alphabet is used instead of the mentioned alphabet.

[0078] Each root is then connected to various other nodes, which may be referred to as child nodes, ending at a final node, which is referred to as a leaf. The set of nodes between a root and a leaf is called a branch. Each branch forms at least one word, with leaf being the last letter of that word. For example, the leftmost branch on column 400 contains the words 'age' and 'ages'. Branches can form proper names, for example, the rightmost branch on column 400, which forms the name ‘Zoe’.

[0079] Column 400 may store multiple conjugations of a word, such as multiple conjugations generated by an inflector (described in more detail below). Multiple conjugations may share one or more common nodes, such as a common root. Each word and/or sheet stored in column 400 may be associated with a metric. The score may indicate the frequency with which the word occurs in the text used to generate column 400. The score for each word may be stored in column 400. The score may be a confidence score for the prediction, such as confidence scores 325 and 335. In some non-limiting embodiments of the present technology, different conjugations of the same word may have the same confidence score.

[0080] Instead of considering each word individually, it may be useful to study the relationship between words in order to improve prediction accuracy. Boost metrics can be defined for words that occur frequently together in the text used to generate graph 400. Boost metrics can be stored separately from column 400. If the first word is recognized in a user utterance using column 400, boost metrics corresponding to that the first word can be used to predict the next word in the user's utterance. For example, some support indicators may support a particular conjugation of the second word that follows the first word if that conjugation of the second word depends on the form of the first word.

[0081] With reference to Figure 5, a diagram of a system for generating a local graph according to non-limiting embodiments of the present technology is illustrated. Client device 210 may store various data for a user of client device 210. For example, client device 210 may store contacts 520, music 530, messages 540 (text, email, etc.), applications 550, and/or other data. Although illustrated as being stored on client device 210, it should be understood that all or parts of contacts 520, music 530, messages 540, and/or applications 550 may be stored outside client device 210, such as in a cloud environment.

[0082] Text may be generated according to contacts 520, music 530, messages 540, applications 550, and/or other data stored by client device 210. For example, the names of all contacts 520 may be extracted, the titles and artists of all music 530 may be extracted, application names 550 can be extracted, and so on. This text may form a body of text used to generate the local graph 215.

[0083] After the text corpus is generated using the data stored on the client device 210, the corpus can be entered into the normalizer 560. The normalizer 560 can be stored and/or executed on the client device 210 and/or on the server. If the normalizer 560 is running on the server, the body of text may be transmitted by the client device 210 to the server. The normalizer 560 may translate any non-text portions of the text corpus into text. Normalizer 560 may convert numbers, abbreviations, slang, and/or local expressions to text. For example, if a text corpus includes the number "24", the normalizer might convert that number to the text "twenty four". In another example, if the text corpus includes "$4.50", the normalizer 560 may convert this to "four dollars, fifty cents". In yet another example, if the body of text includes "12ft", the normalizer 560 may convert this to "twelve feet". Although examples are provided here in English, the normalizer 560 may function for other languages such as Russian. The normalizer 560 may replace the original text in the text corpus with the translated text and/or store the original text in the text corpus and add the translated text to the text corpus.

[0084] After the text corpus has been processed by the normalizer 560, the text corpus can be entered into the inflector 570. Similar to the normalizer 560, the inflector 570 can be stored and/or executed on the client device 210 and/or a separate device such as a server. The inflector 570 can generate alternative forms of text in the body. Inflector 570 can generate alternative word conjugations, alternative spellings of words, and so on. For example, if the text "twenty-four" was received by inflector 570, the inflector may output "twenty four" and "two four". The output of the inflector 570 can be added to the body of the text.

[0085] After the text corpus has been processed by the normalizer 560 and the inflector 570, the local graph 215 can be generated using the text corpus. The local graph 215 may include each word in the text corpus. Local graph 215 may indicate the frequency of occurrence for each word in the text corpus. The local graph 215 can then be used to predict text corresponding to the user's utterance.

[0086] It should be noted that although in the above examples it has been described that the local graph 215 is stored on the client device 210, and the graph 235 is stored on the server 230, these two graphs (local graph 215 and graph 235) can be stored on the same physical medium. information, whether it be a server 230 or a client device 210.

Method (non-limiting embodiment)

[0087] With reference to Figures 6, 7, and 8, a flow diagram of a method 600 is illustrated, with method 600 being implemented in accordance with non-limiting embodiments of the present technology.

Step 605 - Retrieve text stored on the client device

[0088] Method 600 begins at 605. At 605, text and/or other data stored on the client device may be retrieved. The text may include contacts 520, music 530, messages 540, applications 550, playlists, books, calendar data, and/or any other data stored on the client device. While it is described that the text is stored on the client device, other data that is associated with the user of the client device can be retrieved, whether or not that data is stored on the client device, such as user data stored in a cloud environment.

[0089] The retrieved data may come from multiple devices, such as multiple devices associated with the same user. For example, if the user has both a smartphone and a tablet, data can be retrieved from both the smartphone and the tablet.

Step 610 - Apply Text to Normalizer

[0090] At block 610, the text extracted at block 605 may be applied to a normalizer, such as normalizer 560. The normalizer may output normalized text corresponding to the text extracted at block 605. As described above, the normalizer may transform numbers, abbreviations, slang and/or local expressions to text. The normalizer may apply a set of rules and/or a set of features to the text. The normalizer may traverse all of the text retrieved at block 605.

Step 615 - Apply Text to Inflector

[0091] At block 615, the text extracted at block 605 and/or the normalized text generated at block 610 may be input to an inflector, such as inflector 570. The inflector may output the inflectored text. As described above, the inflector may generate alternative forms text such as alternative conjugations, alternative spellings, etc. The inflector may generate alternative word forms in the text received at step 605 and/or the normalized text generated at step 610.

[0092] Depending on the language of the text, steps 610 and/or 615 may be skipped. For example, the normalizer may not be applied to text stored in English. If data in multiple languages is stored on the client device, the text retrieved at block 605 may be clustered by language. The normalizer and/or inflector may be applied to a subset of the clusters based on the language of each cluster.

Step 620 - generate local graph

[0093] At 620, a local graph may be generated using the text extracted at 605, the normalized text generated at 610, and/or the inflected text generated at 615. The local graph may be generated by a client device and/or a server . After the local graph is generated, the local graph can be stored on the client device.

[0094] A local graph may include a tree having nodes and edges connecting those nodes. Each node can correspond to one letter. The first node in each branch of the tree may correspond to the first letter of the word. Subsequent nodes in this branch may correspond to the next letters of the mentioned word. Each branch ends at a node called a leaf. The leaf may contain the last letter of the word formed by the branch. Each leaf can correspond to a word, with the word formed by each letter from the root node to the leaf.

[0095] The graph may indicate the frequency with which each word represented in the graph occurs in the text extracted at block 605, the normalized text generated at block 610, and/or the inflected text generated at block 615. The frequency may be in the form percent. The frequency can be stored as edge weights on the graph.

Step 625 - Receiving a User Statement

[0096] At block 625, a user statement may be received. The user utterance may be received by the client device, such as the microphone of the client device. A user statement can be a word or multiple words. The user statement can be a command, a query, or any other spoken statement. As described above, the trigger word may be spoken to indicate that the user utterance following the trigger word is to be captured.

[0097] After receiving a user utterance, the user utterance may be modified for speech-to-text processing. Filters may be applied to the user utterance, for example, to reduce background noise. The user utterance may be divided into multiple segments, for example, to separate the user utterance into individual words and/or sentences. If multiple speakers are found in the recorded audio data, the user utterance from an individual speaker can be isolated in the audio data.

Step 630 - Use Local Graph to Generate Predictive Text

[0098] At block 630, the local graph generated at block 620 may be used to determine predictive text corresponding to the user utterance received at block 625. The predictive text may be one word or multiple words. The local graph can be used to determine the first confidence score corresponding to the predictive text. The beam search can be performed using a local graph to determine the predictive text.

[0099] As described above, the local graph may indicate the frequency of the prediction word. The first confidence metric may be determined based on the frequency of the prediction word. For example, if the prediction word has a relatively high specified frequency, the first confidence score may be relatively high. In contrast, if the prediction word has a relatively low specified frequency, the first confidence score may be relatively low.

Steps 635 and 640 - Compare First Confidence Score with Threshold Score

[00100] At steps 635 and 640, the first confidence score may be compared to a predetermined threshold confidence score. A predetermined confidence threshold may be set by the operator of the speech-to-text translation system. The predetermined confidence threshold may be a relatively high value. It may be more efficient to rely solely on the predictive text obtained by the local graph, such as local graph 215, instead of providing the prediction by both the local graph and another graph, such as graph 235. If the confidence score for the local graph prediction is high enough, it can be assumed that the forecast for another column will not have a high enough confidence score for it to be selected as the predictive text for output. It can also be assumed that if a local graph prediction has a high enough confidence score, then that local graph prediction is probably correct. In this case, generating a different prediction by a different graph may be redundant, time-consuming, and/or may reduce the accuracy of the speech-to-text system.

[00101] If a determination is made at 640 that the first confidence score is above a confidence threshold, method 600 may terminate at 645. Otherwise, if the first confidence score is below the threshold, method 600 may proceed to step 650 where the predictive the text corresponding to the user's statement will be defined using a different column. In some cases, the speech-to-text translation system may be configured to always determine the second predictive text, in which case steps 635 and 640 may be skipped and method 600 may jump directly from step 630 to step 650.

Step 645 - Outputting First Predictive Text

[00102] At 645, a first predictive text may be output. The first predictive text may be displayed. The first predictive text can be used as input to the application. For example, the first predictive text may be output using an application programming interface (API). If the first predictive text matches the command, the command may be executed by the client device.

Step 650 - Sending the User Statement to the Server

[00103] At step 650, the user statement may be transmitted to a server, such as server 230. The server may be a server that communicates with many other client devices. The server can be a group of servers, such as a cloud application. The server may implement a speech-to-text translation system.

Step 655 - Generate Second Predictive Text

[00104] At 655, a second predictive text may be determined by the server using a graph, such as graph 235. A second confidence metric may be determined according to the second prediction. The graph may be a graph used to generate predictions for multiple client devices. The graph may be generated using a common body of text rather than text specific to the client device such as a local graph. A graph may have a form similar to a local graph. Like a local graph, a graph can include a tree, with each node in this tree corresponding to some letter. The tree can indicate the frequency with which each word in the tree occurs. A beam search can be performed using this graph to determine the second text. The second confidence score can be determined based on the frequency with which the predictive word occurs in the text used to generate the graph.

Steps 660 and 665 - Compare First Confidence Score with Threshold Score

[00105] At steps 660 and 665, the first confidence score may be compared with the second confidence score. If it is determined at 665 that the first confidence score is higher, the first predictive text may then be output at 645. If the second confidence score is higher, the second predictive text may be output at 670.

[00106] In some cases, the first predictive text may still be output even if the second confidence score is higher. The second confidence score can be reduced, for example, by subtracting a predetermined value or applying a formula to the second confidence score. Alternatively, a predetermined threshold may be used, wherein the first predictive text may be output if the first confidence score is within a threshold difference from the second confidence score. For example, if the threshold is five percent, the first confidence score is 87 percent, and the second confidence score is 89 percent, the first predictive text will be output even though the second confidence score is higher.

Step 670 - Outputting Second Predictive Text

[00107] At 670, a second predictive text may be output. Method 600 may then end. The actions performed at block 670 may be similar to those described above with respect to block 645.

[00108] While the implementations described above have been described and shown with reference to specific steps performed in a specific order, it should be understood that these steps may be combined, subdivided, or reordered without deviating from the teachings of the present technology. At least some of the steps mentioned may be performed in parallel or sequentially. Accordingly, the order and grouping of said steps is not a limitation of the present technology.

[00109] It should be clearly understood that not all technical effects mentioned in this document must be achieved in every embodiment of the present technology. For example, embodiments of the present technology may be implemented without providing some of these technical effects to the user, while other embodiments may be implemented with other technical effects provided to the user or without providing any technical effects.

[00110] Some of the above steps, as well as receiving/sending signals, are well known in the art and, as such, have been omitted from parts of this description for simplicity. Signals can be sent/received using optical means (for example, a fiber optic connection), electronic means (for example, using a wired or wireless connection), as well as mechanical means (for example, pressure-based, temperature-based, or any other suitable physical parameter).

[00111] Modifications and improvements to the above-described implementations of the present technology may become apparent to those skilled in the art. The foregoing description is intended to be illustrative and not limiting. Therefore, the scope of the present technology is intended to be limited only by the scope of the appended claims.

Claims (37)

1. A method for translating speech into text, wherein the speech is received by a client device, the method being executable on the client device and comprising: receiving a user statement corresponding to the speech; determining, based on the local graph stored by the client device, the first predictive text corresponding to the user statement and the first confidence indicator corresponding to the first predictive text, wherein the local graph contains a tree, each leaf of this tree corresponds to a word, and the local graph is generated on based on text stored by the client device; transferring a user statement to the server; receiving from the server a second predictive text corresponding to the user statement and a second confidence indicator corresponding to the second predictive text; comparing the first confidence score with the second confidence score; and after determining that the first confidence score is greater than the second confidence score, outputting the first predictive text. 2. The method of claim 1, wherein the first node in each branch of the tree contains the first letter of the word. 3. The method of claim 2, wherein each subsequent node after the first node contains the next letter of the word. 4. The method of claim 1, wherein the first leaf in the tree contains the first conjugation of the word, the second leaf in the tree contains the second conjugation of said word, and the first leaf and the second leaf have a common parent node in the tree. 5. The method of claim 1, wherein each leaf of the tree is associated with a confidence score, wherein the confidence score indicates the frequency with which the word corresponding to the corresponding leaf occurs in text stored by the client device. 6. The method of claim 5, wherein determining the first predictive text comprises defining a leaf of the tree that corresponds to the user statement, wherein the first confidence score contains a confidence score corresponding to that leaf. 7. The method of claim 1, wherein the local graph is generated by: applying the text stored by the client device to the normalizer, thereby generating the normalized text; applying the text stored by the client device and the normalized text to the inflector, thereby generating the inflectored text; and generating a local graph based on the text stored by the client device, the normalized text, and the inflected text. 8. The method of claim 7, wherein the normalizer converts the digits in the text stored by the client device into text. 9. The method of claim 7, wherein the inflector receives a word and generates a plurality of forms of that word. 10. The method of claim 1, wherein the text stored by the client device contains a list of contacts. 11. The method of claim 1, wherein the text stored by the client device matches the music stored by the client device. 12. The method of claim 1, wherein the local graph is personalized for the user of the client device. 13. The method of claim 1, wherein the local graph is a tree of n-grams, with each node in the local graph corresponding to a separate n-gram. 14. The method of translating speech into text, and the speech is received by the client device, and the method is executable on the server and contains: receiving from the client device a user utterance corresponding to the speech; determining, based on the graph, a first predictive text corresponding to the user statement and a first confidence score corresponding to the first predictive text, the graph comprising a tree, each leaf of the tree corresponding to a word; receiving from the client device a second prediction text corresponding to the user utterance and a second confidence indicator corresponding to the second prediction text, wherein the second prediction text was determined by the client device, wherein the second prediction text was determined based on a local graph stored by the client device, wherein the local graph was generated based on the text stored by the client device; comparing the first confidence score with the second confidence score, and after determining that the first confidence score is greater than the second confidence score, transmitting the first predictive text to the client device. 15. The method of claim 14, wherein the first leaf in the tree contains the first conjugation of the word, wherein the second leaf in the tree contains the second conjugation of said word, wherein the first leaf and the second leaf have a common parent node in the tree. 16. The method of claim 14, wherein each leaf of the tree is associated with a confidence score, wherein the confidence score indicates the frequency with which the word corresponding to the corresponding leaf occurs in text stored by the client device. 17. A method for translating speech into text, wherein the speech is received by a client device, the method being executable on the client device and comprising: receiving a user statement corresponding to the speech; determining, based on the local graph stored by the client device, the predictive text corresponding to the user statement, and the reliability indicator corresponding to the predictive text, while the local graph contains a tree, with each leaf of this tree corresponding to a word, and the local graph is generated based on the text, stored by the client device; comparing a confidence score with a threshold confidence score; determining, based on the confidence threshold, whether the user statement should be transmitted to the server; after determining that the confidence score satisfies the threshold confidence score, determining that the user statement should not be transmitted to the server; and output of predictive text.

Images