TW548600B - Method and system for identifying attributes of new words in non-segmented text - Google Patents

Abstract

Embodiments of the present invention provide a method and apparatus for segmenting text by identifying new or rare words in the text. Under the present invention, a sub-string of single characters in the text is identified. For each character in the sub-string, an independent word probability is calculated that indicates the probability that each single character represents a single-character word. The probabilities for all of the characters in the sub-string are combined to form a total probability. If the total probability is below a threshold, the characters in the sub-string are considered to form a single multi-character word. In a further embodiment, the system determines parts of speech for multi-character words that are not found in the dictionary.

Description

54860ο Α7

It may be related to the computer-based method in this article. Furthermore, the present invention relates to the segmentation of the text which will contain new words in a certain language. Word segmentation and formation of linguistic expressions (such as this pass. Same, eight W1 as much as possible) (individual word recognition methods have 1 sub-centred & help to check # 字 和文 ,, this 1 self-cooking solution / , Synthesize the speech from this article, and then search for a group of special characters or phrases. ^ The word segmentation in English text is quite simple, because of spaces and punctuation. But for unsegmented = Japanese such as Japanese or Chinese, the boundaries of the words are implicit and not obvious. That is, the word of the text is not segmented ^ often does not include m punctuation. Therefore, these languages cannot be segmented in the same way as the English word segmentation. In most prior art systems, this text is segmented using a simple hyphenator. These word breakers usually group characters into possible segments and then search the dictionary for that segment. If you find a paragraph in the dictionary, leave it as a possible paragraph in this article. Although this technique is effective for words in the dictionary, it is not effective for new words and rarely used words in the language because they are often not found in the dictionary. Generally, dictionary technology replaces the recognition of a group of characters as a set of single-character characters to form a single "less-used" word. Some systems use statistics-based calculations to expand dictionary-based segmentation to help Identify this " less use ,, word. In this method, use more characters, "less use," word probabilities and some critical comparisons. If this threshold is exceeded, it is recognized as a word. However, because "less used, words rarely appear in language, and the probability is almost always lower than this threshold. Therefore, even -4-this paper size applies the Chinese National Standard (CNS) A4 specification (210X 297 mm)

Binding

548600 A7 B7 V. Description of the invention (2) These expansion systems often fail to have a proper probability of "less use," words. If syntactic analysis is performed in this article, the segmentation system needs to identify rarely used or new words in this article, and recognize that these words may be part of the phonetic. Therefore, the required segmentation system is more capable of recognizing less used words and can recognize fewer or new words in some speech. SUMMARY OF THE INVENTION Embodiments of the present invention provide a method and apparatus for segmenting the text by identifying new or less used words in the text. The present invention recognizes substrings of single characters in the text. The individual word probability is calculated for each character of the substring, and the probability indicates the probability that each single character represents a single character word. The probability of all characters in the substring is combined to form the total probability. If the total probability is below the threshold, the characters in the substring are considered to form a single multi-character word. In another embodiment the system decides on a multi-character portion of speech that cannot be found in the dictionary. In order to achieve this function, the system determines the probability of each character in the word, which indicates the possibility of finding the character at the current position in a word with a multi-character length and a certain part of the voice. For example, for the two-character word "AB '", the system will determine the first probability that the character " Απ appears in the first character of the two-character noun, and the character " Α " is the first in the two-character verb. The second probability of the character and the third probability of the character π A " in the first character of the two-character adjective. The probability of combining these characters in a part of the speech library forms the total probability of each part of the speech. Then each Sum of the total and critical. In the above example, the probability of A " appearing in the first character of the two-character noun will be combined with "B " the probability of appearing in the second character of the two-character noun to obtain" ΑΒΠ is the total probability of nouns. The odds that exceed the threshold are added together as possible parts of the multi-character character. -5- This paper size applies the Chinese National Standard (CNS) Α4 specification (210 X 297 male) (Centimeter) 548600 A7

FI 1 port, optional, simple diagram Figure 1 is suitable for the implementation of the present invention. Figure 2 is a block diagram of a computer system that can be used with caution and target.掌 Palm type of the present invention FIG. 3 is a block diagram of the present invention. FIG. 4 is a detailed block diagram according to the present invention. Shi Wuli, and ^ are examples of Ming Guan's segmentation and identification of this article. Example: An example of a suitable computing environment that can implement the present invention. The plate is too respectful. The box i G G is just an example of a suitable computing environment. It is not intended to limit the use and range of functions. Example operating environment 1 0 0 Any 7 ° component or combination is not a requirement or dependency of the computing environment 1 0 0. It is possible to send the month, other general purpose or special purpose computing environments or frameworks at midnight. 4 The well-known system environment and / or architecture examples used in the present invention are not limited to personal computers, server computers, palmtops or laps. Devices, multi-processor systems, microprocessor-based systems, video converters, programmable consumer electronics devices, network personal computers, mini computers, mainframe computers, decentralized computing environments that include any of the above systems or devices and the like . This month can be described in the general case where the computer executes instructions such as program modules. Program modules usually contain subroutines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The invention can also be implemented using a decentralized computing environment where remote processing devices linked via a communication network perform tasks. In a decentralized computing environment, the program model is -6-t paper size suitable for financial ® S home materials (GNS) Α4 size (210 X 297 mm)

Binding 548600 Five A7 B7, invention description (4) The group can be located in the local and remote computer storage media containing the memory storage device 0 Figure 1 is an example system used to implement the present invention includes a computer in the form of 1 1 0 universal Computing device. The components of the computer 1 10 may include, but are not limited to, a processing unit 120, a system memory 1.3, and a system bus 1 2 1 to couple various system components including the system memory and the processing unit 120. The system bus 1 2 1 can be multiple types of bus architectures, including memory buses or memory controllers, peripheral buses, and local buses using multiple bus architectures. This architecture includes, but is not limited to, industry standard architecture (ISA) buses, microchannel architecture (MCA) buses, enhanced industry standard architecture (EISA) buses, video standards association (VESA) local buses, and peripheral component interconnects (PCI) bus (also known as Mezzanine bus). Computer 1 10 usually contains a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by computer 110 and includes both volatile and non-volatile media, removable and non-removable media. Computer-readable media may include, for example but not limited to, computer storage media and communication media. Computer storage media includes any method or technology that contains volatile and non-volatile, removable and non-removable media and is used to store information such as computer-readable instructions, data instructions, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash-memory or other memory technology, CDROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridge, magnetic tape, magnetic disk storage, or other magnetic storage A storage device, or any other medium that can be used to store desired information and accessible by computer 100. Communication media is usually implemented on modulated data signals such as carrier waves or other transmission media. Computers can be used on this paper. The Chinese standard (CNS) A4 specification (210 X 297 mm) is bound. 548600 5. Description of the invention (5) Reading instructions and data Structure, program module, or its delivery medium. " Modulation data 'and ^ any information ^. X, 777 A signal has one or more of its characteristics changed or the signal will change ^ ^ Λ, the code will be changed. Examples of communication media are not limited to wired media such as wired network or single-line connection, and audio media such as the sound 1 line or wireless media. Any combination of the above should also be included in the scope of computer-readable media. The system memory U0 includes computer storage media in the form of volatile and / or non-volatile memory such as read-only note sounds, π η λ bee dagger, sister (R0M) 13 and random access memory (RAM) 132. The basic input / output system (B10S), which is usually stored in ROM 131, has a basic subroutine that assists in the transfer of resources between components in the computer 1G, such as at startup. Generally, the data and / or programs possessed by the RAM 132 can be immediately accessed and / or used by the processing U12G. For example, non-limiting (FIG. 1 illustrates the operating system 134 'application 135, other program modules 136, and program data 137. The ordering computer 110 may also include other removable / non-volatile, volatile / non-volatile computer storage media. It is only used as Example, figure! Illustrated hard drive M! Read ^ Non-removable non-volatile storage media, removable drive 151 read and write removable, non-volatile drive 152, and optical drive 155 read and write such as CDROM * other optical Removable non-volatile disc 156 of the media. Others that can be used in the example operating environment ^ Removable / non-removable, volatile / non-volatile computer storage media including but not limited to tape E, flash memory card, digital multi-style- Optical discs, digital audio and video tapes, solid-state RAM, solid-state ROM, and the like. The hard disk drive 141 is usually connected by a non-removable memory interface such as interface 1 4 0 and the system bus [2 i, and the disk drive 1 5 1 And CD-ROM drive 1 5 5 usually consists of removable memory such as interface 丨 5 -8-This paper size applies Chinese National Standard (CNS) A4 specification (210 X 297 mm) 548600 A7 B7 V. Description of invention (6) Interface system bus 1 2 1 connected. The above and illustrated in Figure 1 The disc drive and its related computer storage media store computer-readable instructions, data structures, program modules, and other data of the computer 1 1 0. For example, the hard drive 14 i illustrated in FIG. 1 stores the operating system 144, and the application programs 1 4 5, Other program modules i 4 6 and program data 1 4 7. These components can be the same as or different from the operating system 1 3 4, application programs 1 3 5, other program modules 丨 3 6 and program data 1 3 7. System 丨 4 4, application 丨 4 $, other program modules 1 4 6 and program data 1 4 7 The different numbers here at least explain that they are different from each other. Users can use such as keyboard 162, microphone 163, pointing device 161 (such as Input devices such as airbags, trackballs, or touchpads input commands and information into the electronic 1 1 0. Other input devices (not shown) may include joysticks, game control °° disc-shaped antenna antenna, scanner or Similarly, these and other input devices are often connected by a user input interface coupled to the system bus, such as 6o, and a processing order, το 1 2 0, but they can also be connected by other interfaces and bus architectures, such as flat port, Play port, universal serial The bus (usb) monitor 1 91 or other display devices are also connected to the system bus 1 2 1 by an interface such as a video interface 丨 9 〇 In addition to the monitor, the computer can also include other peripheral output devices, such as speakers 197 And printer 196, which can be connected through the output peripheral interface 190. The private computer 110 can use a logical connection in a network environment to act as a remote computer i 02-or multiple remote computers. The remote computer 180 can be personal Computer, palmtop, server, router, network computer, peer device, or other network node, and usually contains many or all of the above related to computer i 10-9-

548600 A7 ___B7 V. Elements of the invention (7 ~~ " ~. The logical connections shown in Figure 1 include a local area network (LAN) 171 and a wide area network (WAN) 173, but other networks can also be included. Such networks Road environment is common in offices, corporate computer networks, intranets and the Internet. When used in a LAN network environment, computers are connected via a network interface or adapter 170 and LAN 171. When used in In a WAN network environment, the computer 11 usually includes a modem 172 or other device for establishing communication with the WAN 173 (such as the Internet). The modem 1 72 can be built-in or external, and the user input interface 16 0 or other appropriate structure can be connected to the system bus 丨 2 1. In a network environment, the program module that displays the computer 11 or a part thereof can be stored in a remote memory storage device. For example (but not limited to) The remote application 185 illustrated in Figure 丨 is located in the remote computer 18o. You must know that the network connection shown is an example, and you can use other equipment to establish a communication link between the computers. Figure 2 < Mobile device 2 〇 〇 The block diagram is an example computing environment. Mobile device 200 Contains microprocessor 202, memory 204, input / output (1 / 〇) component 206, and communication interface 208 for communicating with remote computers or other mobile devices. The above is coupled in a practical example. The components communicate with each other by the appropriate buses 2 and 0. The memory body 204 is implemented in non-volatile memory such as random access memory (RAM) with a battery backup module (not shown) for mobile devices. When the general force of 2000 is turned off, the information of the storage memory 2 will not be lost. Part of the memory 204 is preferably configured as an addressable memory for executing programs, and the other part of the memory 204 is preferably used. In storage, such as the storage of an analog disk drive. 5 Ji Yi 2 0 4 contains the operating system 2 1 2, applications 2 1 4 and object binding.

548600 A7 ____B7 V. Description of the invention (8) Store 2 1 6 When operating, the operating system 2 丨 2 is executed by the processor 202 from the memory 204. In a preferred embodiment, the operating system 212 is a commercial Windows CE brand operating system from Microsoft Corporation. The operating system 2 丨 2 is preferably designed for mobile devices and implements database features that can be used by Application 2M via a set of exposed application program interface methods. Objects in the object store 216 are maintained by the application 2 1 4 and the operating system 2 1 2 at least partially in response to calls to exposed application interfaces and methods. The flood interface 208 represents various devices and technologies that enable the mobile device 2000 to send and receive information. The unit includes wired and wireless modems, satellite receivers and broadcast tuners. The mobile device 2000 can also be connected directly to a computer to exchange data. In this case, the communication interface 208 can be an infrared receiver or a serial or parallel communication connection capable of transmitting outgoing information. The input / output components 206 include a variety of input devices such as a touch screen, keys, rotor, and microphone, and a variety of output devices including an acoustic fort, vibrator, and display. The above devices are only examples, and are not necessarily required for mobile devices 2000. In addition, other input / output devices may be connected to the mobile device 200 or within the scope of the present invention. The embodiment of the present invention provides a method and a device for segmenting this text by identifying rare and new multi-character characters. Another embodiment provides a method and system for recognizing partial non-dictionary phonetics. Fig. 3 is a block diagram of various components according to an embodiment of the present invention. FIG. 4 is a flowchart of a method according to an embodiment of the present invention using the components of FIG. 3. In step 400 of FIG. 4, the hyphenator 300 of FIG. 3 recognizes the adjacent character combination that appears in the small vocabulary record group (also referred to as the dictionary) 300 in the text 300 in this text. Vocabulary Green Group 3 04 is due to the limited amount of grammatical information stored in each character. -11-This paper size applies the Chinese National Standard (CNS) A4 specification (210X 297 mm) 548600 A7 B7 V. Description of the invention (9) 4 Fruit brother green group 3 〇 The number of words included does not have to be very small, in fact in some Example Small Vocabulary The green group 3 04 has a large number of characters. In one embodiment of the present invention, the word breaker 3 02 uses a data structure called a trie to search for words in the small vocabulary green group 304. In clues, the words are not listed in sequence but replaced by state chain representations. Each state represents a character, and contains one or more sub-states, and each sub-state has a character, which occurs after the current state of resentment Feng Yuan in at least one word of the sub-record 1 of the group 3 304. Each state also indicates whether the current character appears in the final character of a character, which is formed by a state chain that processes the current character. With the clue data structure, possible words in the character string such as " ABcd " can be fixed. For example, the system may start in a state such as the character " A ". If the center of mind indicates that the + element " a " is a single word in the small vocabulary record group 3 04, "A" will be recognized as a possible segment of the string. Then the system checks whether there is a character " B " A child state extended from the "A" state of Dongyuan. If there is a "B" substate, check whether the state "B" is the final character of any word. If so, the recognition string " AB " is a possible segmentation. Then the system checks the child status of the state extension. If the character " 一 C " has no sons, then the situation will be extended. The system stops looking for the current chain and starts searching for a new chain starting with the character " B ". Repeatedly start each character of the input string The processing of the new chain is to check that each character may be the beginning of the chain. Once in step 4 0 0, the words stored in the small vocabulary record group 3 0 4 are identified, and the method in FIG. 4 is continued to step 4 02. This hyphenator 30 uses the appropriate naming criterion 3 0 6 to identify whether a word may not be stored in the green group of small words, but it indicates an appropriate name such as a person's name or place name. -12- 548600 A7-______B7 5 Description of the invention (1〇)-The words identified by the small capsule i Jilu group 3 04 and the appropriate naming criterion 3 06 are placed in the character grid 'for access to the large vocabulary record green group 3 1 2 vocabulary review 3 1 〇. The large vocabulary record group 312 contains more vocabulary information than the small vocabulary record group 3 04. In fact, the small vocabulary record green group 3 in the multiple-duration example is built from the large vocabulary record group 3 12 and periodically Reference update. Use Datonghui record group 3 丨 2, in step 4 〇6, vocabulary lookup 3 i 〇 The amount of vocabulary information stored for each word in the grid. This additional information includes the following items, and the origin of the word, whether the word can be used as a proper noun, and other vocabulary and grammatical details of the word. Refer to 3 丨 〇 and send it to the derived word formation 314. In step 408 of FIG. 4, the derived word formation 314 combines the adjacent π segments in the grid to form a large multi-segment word. For example, the derived word formation element 3 丨 4 can end the suffix. The character string, infix character string, and prefix character string are appended, inserted, and prepend to their b segment to form a larger word. In some embodiments, some or all of these derived word formation criteria are broken by hyphenation at step 402. 302 is used instead of the word formation element 3 丨 4 in step 408. However, the advantage of the word formation element 3 丨 4 is that the large vocabulary record group has more information to enter the derived word formation criterion. In addition, the derived word formation Element 3 1 4 can be used to identify and reference segments of other name entities, such as names of people, institutions, and geographic locations, and other appropriate names, and other units such as dates and times.-Derived from the structure of 3 1 4 Larger words and vocabulary information The character grid is added. In most embodiments, the larger word constructed by the derivative word 3 1 4 will not replace the smaller segment, but will be placed in the cell with the smaller segment. The extended word generated by the derivative word 314 The grid is provided to identify the book at step 41 in Figure 4.13-This paper size is suitable for household standard (CNS) μ specifications (21QX297 public love)

Binding 548600 A7

心 新 字 dentifier 320. To recognize the new 320, search for the character in the grid instead of the character in the grid—the ^ ’胄 character recognizer recognizes the string, and the new character recognition 3 2 0 determines the place in the string: the sub-string. Probability of each possibility. In the embodiment, =: is achieved by averaging or increasing the probability of independent words of single-character characters. In the section of this article, the individual characters are single two sub-probabilities, which means that in the embodiment-character: :, :: Γ possibility. The probability of JL Yu Fanma ’s independence is ... N (W〇rd (c)) N (c) (Equation 1) IWP (c) where N (W is the character c in the data and the single N⑷ is the character. The number of times in the data (the number of single characters, word), and the unique character of the character = multi-character 丨. Analysis of Beco's "probability of each child, the use of inspection and analysis results (and count the occurrence of each character And the number of occurrences of each character as an independent word. The profiling data used to form independent word probabilities in the-embodiment 'is not large enough to include all rare π. As long as the data contains often single words. ^ See (Zi Fan, in-embodiment Before using the new word recognizer, calculate the individual word probabilities that appear in all characters. Then store these probabilities in the independent word probabilities M2 and store them in the new word recognizer—word recognizer 32. When the new word recognizes The device 3 2 0 finds a single word το string that is not part of a multi-character word, that is, accesses the independent word probabilities of these characters stored in the storage 322. Then adds the independent word probabilities of the characters in the string Or average to form the total independent word probability of the character string. 548600 A7 B7 V. Description of the invention (12) The character string is totaled The comparison of the independent word probability and the critical probability determines that the character is more likely to form a single word or a single character string. If the total independent word probability is below the threshold, the serial character is considered to form a single word and the new character is added Enter the character grid. If the total independent word probability is above the threshold, the character_column is considered to be a single-character string string. In one embodiment, different numbers of characters in the string use different thresholds. For example, a two-character string string The four-character string is a different threshold. In some embodiments, the character string is limited to two, three, or four characters. For a single-character string with more than two characters, the present invention is implemented For example, determine the total independent word probability of each possible group of the character. For example, if the character string is n ABC ”, the word recognizer 3 2 0 generates the strings π AB”, ', BC ··, and n ABC1, respectively. Probability of total independent words. All strings with total probabilities less than the threshold are added to the character grid as new words. Therefore, if the total probabilities of "BCn & " ABC " are less than the threshold and the total independence of " ΑΒΠ If the word probability is above this threshold, then "BC, f and" ABC "will consider adding characters as possible. The word "&AB;" will not be added to the character grid. In the embodiment of Figs. 3 and 4, before the new word is added to the character grid, step 4 1 2 of Fig. 4 is used to determine the part of it. Speech. Know that a single new word can represent many parts of speech. So part of the speech recognizer 3 2 4 is to recognize all the parts of the speech that the new word may represent. Part of the speech recognizer 3 2 4 uses-part of each character in the word The probability of speech determines the part of speech that a new word may represent. The partial speech probability of a character indicates the possibility of the character in a particular part of the speech, where the length of the character and the position of the character in the character are fixed. For example, some speech probability may refer to a character. " A "probability of the second character in a three-character noun. -15- This paper size applies to Chinese National Standard (CNS) A4 (210 X 297 mm) A7

548600 V. Description of the invention (Some probabilities of speech can be expressed by p, Cat, L, where p is the rate of 'Cat', which is the abbreviation of speech or its category, loc is the position of the character in the word, 'ALen is the word length (word The number of characters). W If the character is in the second character of the four-character verb, it is represented by Pv24, the character is in: the probability of the first character of the character noun is represented by Pnl2 ', and the character is in the third character of the four-character adjective. The probability of a character is represented by Pa34. An embodiment of the present invention restricts part of the speech to be a noun (10, verb (V), and adjective (a), and limits the word length to 2 to 4 characters to limit the calculation of each character. The number of probabilities. Therefore, the probabilistic length attribute can be 2, 3, or 4, and the position attribute can be 1, 2, 3, or 4. The limitation of this group makes the partial voice probability of each character up to 27. In one implementation For example, the probability of each part of speech is determined by the number of words found in the language dictionary. The word has a certain length and part of the speech and the character is in its specific position. Therefore, to determine the character "Pn23" of the character "A", the present invention The number of three-character nouns in the dictionary is A as the second character. Then This count is divided by the number of characters of the character appears to give some voice probability expressed in the following equation:

(Equation 2)

P, cat, loc, len (c) = N (cat, loc, len (c)) N (^) where P, cat, loc, len (c) is part of the speech probability of the character c, N (cat, loc, len (c)) is the number of words and length of words in a specific part, where the character appears in a specific position, and N (c) is the total number of words in which the character appears. In one embodiment, the counting of the partial speech probability is performed only on the main words of the dictionary. A dictionary of 85,135 main words is used in one embodiment. -16-This paper size applies Chinese National Standard (CNS) A4 specification (210X 297 mm) 548600 A7 B7 V. Description of the invention (14) In one embodiment, calculate the language before receiving a piece of text to be processed. Part of the speech probability of each character. Figure 3 shows that these partial voice probabilities are stored in partial voice (POS) probability stores 3 2 6. When the new word recognizer 3 2 0 recognizes the new word, the partial speech recognizer 3 2 4 retrieves the appropriate partial speech probability of each character in the word from the storage POS probability storage 326. For example, for the new word " AB ", some speech recognizers 3 2 4 access part of the speech probabilities Pη12, Pν12 and Pal2 of the character" A π, and part of the speech and probabilities Pη22, Pν22 and Pa22 of the character ΠB, '. . Partial speech probabilities of similar partial speech are then added or averaged to form the total probability of the word. Therefore, the addition of Pη12 of character "A π" and Pη22 of character "quoteB" provides the total probability of the word "ABII". Similarly, the "Pν12" of character "A" and the pν22 of character "Bπ" are added together. Provide the word as π ΑΒΠ as the total probability of the verb. Then compare the total probability with a threshold to determine whether the word may represent one or more parts of the speech. Therefore, the word is the total probability and critical comparison of the noun, and the word is the total probability of the verb. Comparison of the probability and the threshold of the word, and the total probability and the threshold of the adjective. It must be known that although a single threshold was used previously, in other embodiments different thresholds are used for different speeches. The parts of the speech whose total probability exceeds the threshold are considered as possible new words. Part of the voice. So when the part of the speech recognizer 3 2 4 adds the word to the character grid, that is, a part of the voice with a total probability exceeding the threshold is assigned to the word. In one embodiment, if a part of the word has more than the threshold, the right I insert the word into the cell once in each part of the word. Therefore, the total probability of the word being a noun exceeds the threshold and the total probability of the word being a verb exceeds the threshold. Word lattice. In other embodiments, the word applied only to a word lattice, and the highest total probability distribution portion of the present paper -17- applies China National Standard Scale (CNS) Α4 size (210 X 297 mm) 548600 A7 B7

Sub-voice. In an embodiment, if the total partial speech probability does not exceed the threshold, the partial recognition recognizer 3 2 4 inserts the word into the character grid and expresses the word as a noun. Although the method of deciding the sound of a part of a new character is combined with the method for recognizing a new character and the method of the present invention, the two aspects of the present invention need not be implemented together. Therefore, the method of recognizing new characters can be used without the need to identify new characters, and the method of removing lambda to recognize part of the speech of new characters can be used with any method that recognizes new characters. Another technique is to use the character grid generated by the partial speech recognizer 324 for syntactic analysis. In step 4 1 4 of FIG. 4, the syntactic analysis is performed by using the extended character grid. In: For example, use the bottom-up analysis chart (bottom_up chart to analyze syntactic parsing from = τ: applying phrases to build increasing phrases, perform syntactic parsing broadly: and larger phrases' syntax parser 3 16 quote grammar Rule check: Mutual vocabulary representation to determine the combination of larger words or phrases. Example of using a binary grammar to describe a concatenated word or phrase with = 'Knot' Implementation of a parser 3 16 Syntax analysis performed Consider the extended word Wan. Segment. Restrict the parser to only combine the original input text, the ten characters with a knife, and the eight adjacent words, and the final analysis continues to the entire input text. The knife generation is related to the two overlapping segments or -The group does not represent all new valid parsing.. — 子 _ &lt; Knife # In the parsing again, the parser 3 16 will decide whether any new words of the new word are based on the partial speech recognizer 3 = 32 ^ to identify the speech forming part Effective analysis is only worthwhile. If the new word does not form part Hachiman <New Han Shao Branch identified as the new word in this article. In the present invention, the knife is effectively analyzed, and the new word recognizer-18-

548600 V. Description of the invention (about half of the characters of π 3 2 0 that are not formed are discarded when they are not valid. Choose the new character to identify E32g. The _ is therefore analyzing the politics: analysis', and in many cases the formation of this article is: There is a point in the test—if no new word recognizer 32 () provides ^ important. About 2 i% of 3 i 6 parsed sentences cannot be parsed. 'Syntax parser in some embodiments, syntax parser 3 丨 6 "Generate multi-analysis" each represents the different effective segmentation of the input text. In "solid effective syntax", the parsing syntax is sent to the logical form generator 318, which is related to the two examples. "Ran uses this semantic_, in The semantics of the ^ method is correct for the input string. It is called Zhea. The most likely choice is 416. $ &quot; pound. This semantic identification is shown in the steps of Figure 4 and is described with reference to a specific embodiment. It will be understood that the form and details of the official can be understood without deviating from the spirit and scope of the present invention. Revision-19- This paper size applies to the Chinese National Standard (CNS) A4 specification (210X297 mm)

Claims (1)

090124532 Patent Application Chinese Application for Patent Scope Replacement (May 1992) VI. Scope of Patent Application 1. A method for using a non-segmented language method. The method includes: identifying a single character string of an input character string Column; determine the independent word probability of each character, the probability table of the possibility of the element word; the square string segmentation of the character is a single character to determine the single character display single character string combined with the independence of each character in the single character string Word probability, the total independent word probability of the meta string; and if the total independent word probability is less than the critical value, the single word list is 0. 2. The method according to the first scope of the patent application, where the identification of the rotating character string is early. A character string contains a vocabulary lookup to group the character groups of the input character string into a multi-character word, and search for a single character string that is not included in the input character string ^. Sub 70 3. If the method of the second item of the patent scope is requested, the step of identifying a single character string in the input character string is additionally included in the search input character string ^ A single character queue that is not included in the character string Before, execute it to multi-character characters. 4. The method of reducing the number of items, such as the third item in the scope of patent application, in which the step of identifying the earlier character string in the .f string is also included in the search rotation character string. The first thing to do is to implement the appropriate multi-character characters. Decide 4 to discern 5. The method of item 1 of the scope of patent application, which decides ^ 1, &gt;, the independent full probability includes the number of times a character in the data appears as a single character, divided by the number of times The number of occurrences of the character in the data. And this paper size applies the Chinese National Standard (CNS) A4 specification (210X 297 public shame :) 548600 A8 B8 C8 D8 6. Scope of patent application 6. For the method of applying for item 1 of the patent scope, the combination of independent word probability includes the average independent word probability. 7. The method according to item 1 of the scope of patent application, wherein combining the independent word probabilities includes adding the independent word probabilities. 8. If the method of the first scope of patent application, the method further includes allocating part of the speech to a single word representing a single-character string. 9. The method of claim 8 in the patent application scope, wherein allocating partial speech includes: determining a partial speech probability of each character in a single character, the probability indicates that according to the single word length and the position of the character in the single character, the character is Possibility of words in a specific part of speech; combining all part of speech probabilities of a single character to determine the total part speech probability; and if the total part speech probability exceeds a critical value, allocating part of the speech related to the total part speech probability to the single part word. 10. The method of claim 9 in the scope of patent application, wherein the partial speech probability of determining a character includes: counting a word representing a specific part of speech in a dictionary, the speech having a single word length and containing the character and the same position of the single word To provide a partial speech count; and divide the partial speech count by the number of words in the dictionary containing the character to form a partial speech probability of the character. 11. The method of item 9 of the patent application, wherein determining part of the speech probability includes retrieving part of the previously determined speech probability from memory. -2- This paper size applies to China National Standard (CNS) A4 specifications (210X 297 mm) A BCD 548600 6. Application for patent scope 12. If the method of item 9 of the patent scope is applied, it also includes the total part of all voice The probability of speech does not exceed the threshold, and a single word is used as a noun. 13. If the method of the scope of application for the first item of the patent, further includes: identifying the early character sub-strings in the early character string; combining the individual word probabilities of each character in the single-character sub-string to determine the single-character sub-string The total independent word probability of the column; and if the total independent word probability is less than the critical value, the single-character substring is represented by a single word. 14. A method for recognizing part of the sound of a character, the method comprising: determining a character probability of each character in the character, the probability indicating a possibility of the character appearing in the character of a specific part of the speech; combining all characters in the character; The probabilities of the characters of the yuan form the total partial speech probability; and if the total partial speech probability exceeds the threshold, the partial speech is represented by the word. 15. The method of claim 14 in the scope of patent application, wherein the probability of determining a character includes: counting all words in a dictionary representing a specific part of speech, the speech length is the same as the word and the character and its position in the word Same to generate a partial phonetic word count; and divide the partial phonetic word count by the number of words with that character in the dictionary to form a character probability. 16. For the method of applying for item No. 14 in the scope of patent application, the steps of determining the probability of characters, combining the probability of characters, and expressing characters in parts of speech are in multi-part languages. -3- This paper applies Chinese National Standard (CNS) Α4. Specifications (210 X 297 mm) 548600 Repeated. 17 · —A kind of used in—Fu Fangdian Yin V. 彡 a-丄 禾 刀 奴 ~ Again &lt; a sub-string of strings The system contains: 卞 &lt; 糸 said,-word breaker, using vocabulary record group identification Characters in the character string; ^ 4 m recognizer 'uses another character that determines a single character in the character string that was not crying by a hyphen ^ · in the character, and recognizes the newness in the character string. Word placement-18. If the system of the 17th area of the patent application is applied, another part, the speech recognition of each new word recognized by the new word recognizer "identification 19. If the system of the 18th area of the patent application is applied, some of the speech recognition Use a set of character probabilities to identify parts of a new word: the single character in the spear word is related, and the probability of each character refers to the same degree (the position in the word is the same as the position of the new word in Kazaki, which may indicate a specific Partial speech. This 20 · = system of item 19 of the scope of patent application, in which part of the speech recognizer is the probabilities of all characters in Houxinfeng to generate the total part of the speech probability. 21 · ^ Item 20 of the scope of patent application System, in which part of the speech recognizer program compares the total probabilities of bones and the critical value to determine whether part of the speech related to the total part of speech depends on the new word. The system also includes the syntactic parser to detect the root of the grammar and the new parser to generate the character syntactic parsing of the character. The system of claim No. 22 of the patent scope, in which the parser is excluded by the syntactic parser. New words. 24. The system of item 17 in the scope of patent application, in which the characters form a single character guide &lt; the probability is obtained by dividing the count of single characters in the data by the driving standard of 4-1 (CNS) A4 size (210X297 mm) Binding 8 8 8 8 ABCD 548600 ^ r ^ VI. Scope of Patent Application The number of characters in this document is determined. 25. — A computer-readable medium with computer-executable instructions to perform the following steps: Recognize a single character string in the input character string; Determine the independent word probability of each character, which indicates that the character The possibility of single-character characters appearing; Combine the independent word probabilities of each character in the single-character string to determine the total independent word probability of the single-character string; and if the total independent word probability is less than the threshold, use a single character to represent the single-character string J1J 〇 This paper size is applicable to China National Standard (CNS) A4 (210X 297mm)