WO2021129410A1 - 文本处理方法及装置 - Google Patents

文本处理方法及装置 Download PDF

Info

Publication number
WO2021129410A1
WO2021129410A1 PCT/CN2020/135633 CN2020135633W WO2021129410A1 WO 2021129410 A1 WO2021129410 A1 WO 2021129410A1 CN 2020135633 W CN2020135633 W CN 2020135633W WO 2021129410 A1 WO2021129410 A1 WO 2021129410A1
Authority
WO
WIPO (PCT)
Prior art keywords
word
candidate
wrong
words
candidate word
Prior art date
Application number
PCT/CN2020/135633
Other languages
English (en)
French (fr)
Inventor
祝官文
刘杰
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2021129410A1 publication Critical patent/WO2021129410A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G06F16/3331—Query processing
    • G06F16/334—Query execution
    • G06F16/3344—Query execution using natural language analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/232—Orthographic correction, e.g. spell checking or vowelisation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/237—Lexical tools
    • G06F40/242—Dictionaries
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/279—Recognition of textual entities
    • G06F40/284—Lexical analysis, e.g. tokenisation or collocates
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent

Definitions

  • This application relates to the field of natural language processing, and more specifically, to a text processing method and device.
  • Artificial intelligence is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results.
  • artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a similar way to human intelligence.
  • Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.
  • Text error correction is to perform error detection on the original text and correct the errors according to natural language processing technology.
  • One is to judge whether the input query word is correct based on the dictionary, detect the wrong word, and generate the candidate word corresponding to the wrong word, and use the candidate word to correct the wrong word.
  • the other is to extract contextual semantic information through language models, detect wrong words, and generate candidate words corresponding to the wrong words, and use the candidate words to correct the wrong words.
  • the above-mentioned methods are all time-consuming, and have high requirements on the computing power or communication delay of the device, which greatly affects the fast implementation of end-to-end text error correction.
  • the present application provides a text processing method and device, which can increase the generation speed of candidate words and reduce the time consumption of text error correction.
  • a text processing method which includes: obtaining the text to be processed; performing error detection processing on the text to be processed to obtain the wrong words in the text to be processed; and determining the corresponding error words according to the pre-stored wrong word candidate word database
  • the candidate word set of the wrong word, the wrong word candidate word library is used to indicate the candidate word set corresponding to multiple wrong words; determine the target candidate word corresponding to the wrong word in the candidate word set corresponding to the wrong word; according to the target candidate word pair corresponding to the wrong word Correct the wrong words.
  • the text to be processed may be an optical character recognition (OCR) output text, or it may be a text input by a user.
  • OCR optical character recognition
  • the text input by the user may include content posted in a social network, or may be content entered in a search box of a search engine, and so on.
  • the text to be processed can be any text that needs to be corrected, and this application does not limit the specific form of the text to be processed.
  • the wrong words may include non-words.
  • non-word error detection may be performed on the text to be processed based on the first preset vocabulary to obtain the non-word in the text to be processed.
  • Non-words refer to words that do not exist in the first preset thesaurus.
  • the first preset lexicon may be an English dictionary.
  • Non-words are words that do not exist in English dictionaries, for example, wasld.
  • the wrong word may include the wrong true word.
  • the true word error detection of the text to be processed can be performed based on the language model to obtain the false true word in the text to be processed.
  • the language model may be a statistical language model.
  • the language model can also be a neural network model.
  • the error detection processing may include detecting non-word errors or detecting true word errors, and may also include detecting both non-word errors and true word errors.
  • the wrong word candidate word database may be a wrong word candidate word database generated offline.
  • the candidate words corresponding to the wrong word can be quickly generated, the calculation amount is small, the time consumption of text error correction is reduced, and the real-time nature of text processing is ensured .
  • determining the target candidate word corresponding to the wrong word in the candidate word set corresponding to the wrong word includes: according to the difference between the candidate word and the wrong word in the candidate word set corresponding to the wrong word Similarity and the perplexity of the candidate words in the candidate word set corresponding to the wrong word Score the candidate words in the candidate word set corresponding to the wrong word, where the perplexity of the candidate words in the candidate word set corresponding to the wrong word is used to indicate the wrong word The likelihood of the candidate words in the corresponding candidate word set appearing in the text to be processed; the candidate word with the highest score in the candidate word set corresponding to the wrong word is determined as the target candidate word corresponding to the wrong word.
  • the perplexity of the candidate words in the candidate word set corresponding to the wrong word can be scored by the language model.
  • the score corresponding to each candidate word can be obtained by weighting the scores corresponding to the above items, that is, the weights are set for the scores corresponding to each item.
  • the weight can be preset or obtained through training.
  • the candidate words are scored by using the similarity between the candidate word and the wrong word and the degree of confusion of the candidate word, while considering the similarity between the wrong word and the candidate word and the semantic information of the text to be processed, Better candidate words can be obtained, which improves the accuracy of text error correction.
  • the similarity between the candidate words in the candidate word set corresponding to the wrong word and the wrong word includes the morphology between the candidate words in the candidate word set corresponding to the wrong word and the wrong word Similarity and the edit distance between the candidate word in the candidate word set corresponding to the wrong word and the wrong word.
  • Morphological similarity is used to measure the similarity of two words in morphological characteristics. For example, based on the maximum common character string and character similarity, the morphological similarity of the candidate words in the candidate word set corresponding to the wrong word and the wrong word can be judged, that is, the score corresponding to the morphological similarity can be obtained.
  • the wrong word includes a non-word
  • the wrong word candidate word database includes a non-word candidate word database
  • the non-word candidate word dictionary includes candidate word sets corresponding to multiple common subwords, and the candidate word corresponding to the wrong word is determined according to the pre-stored error word candidate word database
  • the set includes: generating common subwords corresponding to non-words, where the similarity between the common subwords and non-words meets a preset condition; according to the non-word candidate word library, the candidate word set corresponding to the common subwords is determined as non-words.
  • the set of candidate words corresponding to the word includes: generating common subwords corresponding to non-words, where the similarity between the common subwords and non-words meets a preset condition; according to the non-word candidate word library, the candidate word set corresponding to the common subwords is determined as non-words.
  • the set of candidate words corresponding to the word includes: generating common subwords corresponding to non-words, where the similarity between the common subwords and non-words meets a preset condition; according to the non-word candidate word
  • the similarity between the common subword and the non-word may include the edit distance between the common subword and the non-word.
  • the preset condition may be that the maximum edit distance between the common subword and the non-word is less than a preset value.
  • the preset value may be 2. It should be understood that this is only for illustration, and the similarity between the common subword and the non-word may also include other forms of similarity.
  • the similarity between the common subword and the non-word includes the common subword and the non-word. Morphological similarity between non-words. This application does not limit the method for determining the similarity between the common subword and the non-word.
  • the non-word candidate word database can be generated offline.
  • the candidate word set of the non-word is determined by the candidate word set corresponding to the common subword.
  • the time consumption of text error correction can be significantly reduced;
  • the candidate word set corresponding to multiple common subwords in the non-word candidate word library is based on the candidate words corresponding to the multiple common subwords and the multiple common subwords. The similarity between the two is determined.
  • the similarity between multiple common subwords and multiple candidate words corresponding to non-words may include edit distances and/or common character strings between the multiple common subwords and candidate words corresponding to the multiple common subwords. That is, the candidate words corresponding to the common subword can be determined based on the minimum edit distance and/or the maximum common character string.
  • similarity may also be other forms of similarity, such as character similarity.
  • the wrong word includes the wrong true word
  • the wrong word candidate word database includes a wrong true word candidate word database
  • the wrong true word candidate word database includes multiple incorrect true word correspondences.
  • the candidate word set corresponding to multiple false true words in the false true word candidate word database is based on the candidate words corresponding to the multiple false true words and the multiple false true words The similarity between is determined.
  • the similarity between multiple false true words and candidate words corresponding to multiple false true words may include edit distances and/or common character strings between multiple false true words and candidate words corresponding to multiple false true words . That is, the candidate word corresponding to the wrong true word can be determined based on the minimum edit distance and/or the maximum common character string.
  • similarity may also be other forms of similarity, such as character similarity.
  • correcting the wrong word according to the target candidate word corresponding to the wrong word includes: judging the morphological similarity between the target candidate word corresponding to the wrong word and the wrong word; When the similarity is higher than or equal to the preset threshold, the target candidate word corresponding to the wrong word is used as the correction result of the wrong word.
  • the target candidate word corresponding to the wrong word is judged by the morphological similarity of the wrong word, and the target candidate word whose morphological similarity is higher than or equal to the preset threshold is used to correct the wrong word, which can avoid introducing new Mistakes, for example, correcting the original recognition to the wrong word.
  • a text processing device including: an acquisition unit and a processing unit.
  • the obtaining unit is used to obtain the to-be-processed text.
  • the processing unit is used to: perform error detection processing on the text to be processed to obtain the wrong words in the text to be processed; determine the candidate word set corresponding to the wrong word according to the pre-stored wrong word candidate word database, and the wrong word candidate word database is used for indication
  • the candidate word set corresponding to multiple wrong words; the target candidate word corresponding to the wrong word is determined in the candidate word set corresponding to the wrong word; the wrong word is corrected according to the target candidate word corresponding to the wrong word.
  • the processing unit is configured to: according to the similarity between the candidate words in the candidate word set corresponding to the wrong word and the wrong word, and the difference between the candidate words in the candidate word set corresponding to the wrong word Perplexity scores the candidate words in the candidate word set corresponding to the wrong word, where the perplexity of the candidate words in the candidate word set corresponding to the wrong word is used to indicate that the candidate words in the candidate word set corresponding to the wrong word appear in the text to be processed
  • the candidate word with the highest score in the candidate word set corresponding to the wrong word is determined as the target candidate word corresponding to the wrong word.
  • the similarity between the candidate words in the candidate word set corresponding to the wrong word and the wrong word includes the morphology between the candidate words in the candidate word set corresponding to the wrong word and the wrong word Similarity and the edit distance between the candidate word in the candidate word set corresponding to the wrong word and the wrong word.
  • the wrong word includes a non-word
  • the wrong word candidate word database includes a non-word candidate word database
  • the non-word candidate word vocabulary includes candidate word sets corresponding to multiple common subwords
  • the processing unit is used to: generate common subwords corresponding to the non-word, where the common The similarity between sub-words and non-words satisfies the preset condition; the candidate word set corresponding to the common sub-word is determined as the candidate word set corresponding to the non-word according to the non-word candidate word library.
  • the candidate word set corresponding to multiple common subwords in the non-word candidate word library is based on the multiple common subwords and the candidate words corresponding to the multiple common subwords. The similarity between the two is determined.
  • the wrong word includes the wrong true word
  • the wrong word candidate word database includes the wrong true word candidate word database
  • the wrong true word candidate word database includes multiple incorrect true word correspondences.
  • the candidate word set corresponding to multiple false true words in the false true word candidate word database is based on the candidate words corresponding to multiple false true words and multiple false true words The similarity between is determined.
  • the processing unit is used to: determine the morphological similarity between the target candidate word corresponding to the wrong word and the wrong word; when the morphological similarity is higher than or equal to the preset threshold Next, take the target candidate word corresponding to the wrong word as the correction result of the wrong word.
  • a text processing device in a third aspect, includes: a memory for storing a program; a processor for executing the program stored in the memory, and when the program stored in the memory is executed by the processor
  • the processor is configured to execute the text processing method in the foregoing first aspect or any one of the first aspects.
  • a computer-readable medium stores program code for device execution, and the program code includes text used in the above-mentioned first aspect or any one of the first aspects. Approach.
  • a computer program product includes: computer program code, which when the computer program code runs on a computer, causes the computer to execute the first aspect or any one of the first aspects above.
  • a text processing method in one implementation is provided.
  • the above-mentioned computer program code may be stored in whole or in part on a first storage medium, where the first storage medium may be packaged with the processor, or may be packaged separately with the processor.
  • first storage medium may be packaged with the processor, or may be packaged separately with the processor.
  • a chip in a sixth aspect, includes a processor and a data interface, the processor reads instructions stored in a memory through the data interface, and executes any one of the above-mentioned first aspect or the first aspect The text processing method in the implementation mode.
  • the chip may further include a memory in which instructions are stored, and the processor is configured to execute instructions stored on the memory.
  • the processor is used in the text processing method in the foregoing first aspect or any one of the first aspects.
  • FIG. 1 is a schematic diagram of an application scenario of natural language processing provided by an embodiment of the present application
  • FIG. 2 is a schematic diagram of another application scenario of natural language processing provided by an embodiment of the present application.
  • Fig. 3 is a schematic diagram of a natural language processing related device provided by an embodiment of the present application.
  • FIG. 4 is a schematic diagram of a system architecture provided by an embodiment of the present application.
  • Fig. 5 is a schematic diagram of text processing according to a CNN model provided by an embodiment of the present application.
  • Fig. 6 is another schematic diagram of text processing according to the CNN model provided by the embodiment of the present application.
  • FIG. 7 is a schematic diagram of the hardware structure of a chip provided by an embodiment of the present application.
  • FIG. 8 is a schematic diagram of an application scenario provided by an embodiment of the present application.
  • FIG. 9 is a schematic flowchart of a text processing method provided by an embodiment of the present application.
  • FIG. 10 is a schematic flowchart of another text processing method provided by an embodiment of the present application.
  • FIG. 11 is a schematic block diagram of a text processing apparatus provided by an embodiment of the present application.
  • Fig. 12 is a schematic block diagram of a text processing apparatus provided by an embodiment of the present application.
  • Figure 1 shows a natural language processing system that includes user equipment and data processing equipment.
  • user equipment includes smart terminals such as mobile phones, personal computers, or information processing centers.
  • the user equipment is the initiator of natural language data processing, and as the initiator of requests such as language question and answer or query, usually the user initiates the request through the user equipment.
  • the above-mentioned data processing device may be a device or server with data processing functions such as a cloud server, a network server, an application server, and a management server.
  • the data processing equipment receives the query sentence/voice/text question sentence from the smart terminal through the interactive interface, and then performs machine learning, deep learning, search, reasoning, decision-making and other language through the memory of the data storage and the processor of the data processing. data processing.
  • the memory in the data processing device can be a general term, including a database for local storage and storing historical data.
  • the database can be on the data processing device or on other network servers.
  • the user equipment can receive instructions from the user. For example, the user equipment can receive a piece of text input by the user, and then initiate a request to the data processing device, so that the data processing device responds to the user equipment
  • a piece of text executes natural language processing applications (for example, text classification, text sequence labeling, translation, etc.), so as to obtain the processing results of the corresponding natural language processing application for the piece of text (for example, text classification, text sequence labeling, translation, etc.) .
  • the user equipment may receive the text to be processed input by the user, and then initiate a request to the data processing device, so that the data processing device classifies the text to be processed, so as to obtain a classification result for the text to be processed.
  • the classification result can refer to the user's semantic intention indicated by the text to be processed, for example, the user's intention to indicate singing, setting time, and opening navigation; or, the classification result can also be used to indicate the user's emotional classification result, such as ,
  • the classification result may indicate that the user sentiment corresponding to the text to be processed is classified as depressed, happy, or angry.
  • the data processing device in FIG. 1 can execute the text processing method of the embodiment of the present application.
  • Figure 2 shows another natural language processing system.
  • the user equipment is directly used as a data processing device.
  • the user equipment can directly receive input from the user and process it directly by the hardware of the user equipment itself.
  • Figure 1 is similar, and you can refer to the above description, which will not be repeated here.
  • the user equipment can receive instructions from the user, and the user equipment itself classifies the text to be processed to obtain the classification result of the text to be processed.
  • the user equipment can receive instructions from the user.
  • the user equipment can receive a piece of text input by the user, and then the user equipment itself executes a natural language processing application (for example, text Classification, text sequence labeling, translation, etc.), so as to obtain the processing result of the corresponding natural language processing application for the piece of text (for example, text classification, text sequence labeling, translation, etc.).
  • a natural language processing application for example, text Classification, text sequence labeling, translation, etc.
  • the user equipment itself can execute the text processing method of the embodiment of the present application.
  • Fig. 3 is a schematic diagram of a natural language processing related device provided by an embodiment of the present application.
  • the user equipment in FIG. 1 and FIG. 2 may specifically be the local device 130 or the local device 120 in FIG. 3, and the data processing device in FIG. 1 may specifically be the execution device 110 in FIG. 3, where the data storage system 150 may To store the to-be-processed data of the execution device 110, the data storage system 150 may be integrated on the execution device 110, or may be set on the cloud or other network servers.
  • the processors in Figure 1 and Figure 2 can perform data training/machine learning/deep learning through neural network models or other models, and use the data finally trained or learned models to process the input text to be processed, so as to obtain the to-be-processed text Text processing result.
  • a neural network can be composed of neural units.
  • a neural unit can refer to an arithmetic unit that takes x s and intercept 1 as inputs.
  • the output of the arithmetic unit can be:
  • s 1, 2,...n, n is a natural number greater than 1
  • W s is the weight of x s
  • b is the bias of the neural unit.
  • f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal.
  • the output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function.
  • a neural network is a network formed by connecting multiple above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit.
  • the input of each neural unit can be connected with the local receptive field of the previous layer to extract the characteristics of the local receptive field.
  • the local receptive field can be a region composed of several neural units.
  • Deep neural network also known as multi-layer neural network
  • the DNN is divided according to the positions of different layers.
  • the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer.
  • the first layer is the input layer
  • the last layer is the output layer
  • the number of layers in the middle are all hidden layers.
  • the layers are fully connected, that is to say, any neuron in the i-th layer must be connected to any neuron in the i+1th layer.
  • DNN looks complicated, it is not complicated in terms of the work of each layer. Simply put, it is the following linear relationship expression: among them, Is the input vector, Is the output vector, Is the offset vector, W is the weight matrix (also called coefficient), and ⁇ () is the activation function.
  • Each layer is just the input vector After such a simple operation, the output vector is obtained Due to the large number of DNN layers, the coefficient W and the offset vector The number is also relatively large.
  • DNN The definition of these parameters in DNN is as follows: Take coefficient W as an example: Suppose in a three-layer DNN, the linear coefficients from the fourth neuron in the second layer to the second neuron in the third layer are defined as The superscript 3 represents the number of layers where the coefficient W is located, and the subscript corresponds to the output third-level index 2 and the input second-level index 4.
  • the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as
  • Convolutional neural network (convolutional neuron network, CNN) is a deep neural network with a convolutional structure.
  • the convolutional neural network contains a feature extractor composed of a convolutional layer and a sub-sampling layer.
  • the feature extractor can be regarded as a filter.
  • the convolutional layer refers to the neuron layer that performs convolution processing on the input signal in the convolutional neural network.
  • a neuron can be connected to only part of the neighboring neurons.
  • a convolutional layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. Neural units in the same feature plane share weights, and the shared weights here are the convolution kernels.
  • Sharing weight can be understood as the way of extracting image information has nothing to do with location.
  • the convolution kernel can be initialized in the form of a matrix of random size. In the training process of the convolutional neural network, the convolution kernel can obtain reasonable weights through learning. In addition, the direct benefit of sharing weights is to reduce the connections between the layers of the convolutional neural network, and at the same time reduce the risk of overfitting.
  • the neural network can use an error back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, forwarding the input signal until the output will cause error loss, and the parameters in the initial neural network model are updated by backpropagating the error loss information, so that the error loss can be converged.
  • the backpropagation algorithm is a backpropagation motion dominated by error loss, and aims to obtain the optimal parameters of the neural network model, for example, the weight matrix.
  • NLP Natural language processing
  • Natural language is human language
  • natural language processing is the processing of human language.
  • Natural language processing is a process of systematic analysis, understanding and information extraction of text data in an intelligent and efficient way.
  • automatic summarization automatic summarization
  • machine translation MT
  • NER Named entity recognition
  • RE relation extraction
  • RE information extraction
  • IE information extraction
  • sentiment analysis speech recognition
  • speech recognition question answering, topic segmentation, etc.
  • the language model is the basic model in NPL.
  • LM can infer the probability of unknown words based on existing information (such as text information such as words that have appeared in the context). It can also be understood that LM is used To calculate the probability model of a sentence.
  • the language model is the probability distribution of the natural language text sequence, which represents the possibility of the existence of a certain length of a certain sequence of text.
  • the language model predicts what the next word will be based on the context. Since there is no need to manually label the corpus, the language model can learn rich semantic knowledge from an unlimited large-scale corpus.
  • Language models mainly include statistical language models and neural network language models.
  • the neural language model can use the semantic information in the text to provide rich background knowledge for the prediction of the next word and the ranking of candidate words.
  • OCR mainly takes a scanner or camera as input, detects text information on it, recognizes text information in the target area through a character recognition algorithm, and converts it into a text format for subsequent word processing.
  • the OCR system includes image preprocessing, image binarization, noise removal, image enhancement, image correction, layout analysis, character cutting, character recognition, layout restoration, post-processing and other steps.
  • the text processing method in the embodiment of the present application is applied in the post-processing step.
  • an embodiment of the present application provides a system architecture 200.
  • the data collection device 260 is used to collect training data.
  • the training data in the embodiment of the present application may be training text of a training text processing model.
  • the data collection device 260 stores the training data in the database 230, and the training device 220 trains to obtain the target model/rule 201 based on the training data maintained in the database 230 (that is, a type of text processing in the embodiment of this application). model).
  • the target model/rule 201 can be used to implement the text processing method provided in the embodiment of the present application, that is, the text to be processed is processed through relevant preprocessing (the preprocessing module 213 and/or the preprocessing module 214 can be used for processing). Input the target model/rule 201 for processing, and then the processing result corresponding to the target task executed by the text processing model can be obtained.
  • the text processing model can be a text error correction model, and the text to be processed is input into the target model/rule 201 (that is, the text processing model of this application) for text error correction processing to obtain The error correction text of the text to be processed.
  • the text processing model can be a text translation model.
  • the text to be processed is input into the target model/rule 201 (the text processing model of this application) for translation processing, and the translation of the text to be processed can be obtained. text.
  • the target model/rule 201 is obtained by training the original processing model. It should be noted that in actual applications, the training data maintained in the database 230 may not all come from the collection of the data collection device 260, and may also be received from other devices.
  • the training device 220 does not necessarily perform the training of the target model/rule 201 completely based on the training data maintained by the database 230. It may also obtain training data from the cloud or other places for model training. The above description should not be used as a reference to this application. Limitations of the embodiment. It should also be noted that at least part of the training data maintained in the database 230 may also be used to execute the process of setting 210 to process the text to be processed.
  • the target model/rule 201 trained according to the training device 220 can be applied to different systems or devices, such as the execution device 210 shown in FIG. 4, which can be a terminal, such as a mobile phone terminal, a tablet computer, notebook computers, augmented reality (AR)/virtual reality (VR), in-vehicle terminals, etc., can also be servers or clouds.
  • the execution device 210 shown in FIG. 4 can be a terminal, such as a mobile phone terminal, a tablet computer, notebook computers, augmented reality (AR)/virtual reality (VR), in-vehicle terminals, etc., can also be servers or clouds.
  • the execution device 210 is configured with an input/output (input/output, I/O) interface 212 for data interaction with external devices.
  • the user can input data to the I/O interface 212 through the client device 240.
  • the input data in this embodiment of the present application may include: text to be processed.
  • the preprocessing module 213 and/or the preprocessing module 214 are used for preprocessing according to the input data received by the I/O interface 212.
  • the preprocessing module 213 and the preprocessing module 214 may not be provided (or There is only one preprocessing module), and the calculation module 211 is directly used to process the input data. It should be noted that the preprocessing module 213 or the preprocessing module 214 can preprocess all input data, or can preprocess part of the input data.
  • preprocessing module 113 and/or the preprocessing module 214 may also be trained in the training device 220.
  • the calculation module 211 may be used to perform calculations and other related processing on the input data from the preprocessing module 213 or the I/O interface 212 according to the target model/rule 201 described above.
  • the execution device 210 When the execution device 210 preprocesses input data, or when the calculation module 211 of the execution device 210 performs calculations and other related processing, the execution device 210 can call data, codes, etc. in the data storage system 250 for corresponding processing. , The data, instructions, etc. obtained by corresponding processing may also be stored in the data storage system 250.
  • the I/O interface 212 feeds back the processing results (such as error correction results, translation results, etc.) to the client device 240.
  • processing results such as error correction results, translation results, etc.
  • the training device 220 can generate a target model/rule 201 corresponding to the downstream system for different downstream systems, and the corresponding target model/rule 201 can be used to achieve the above goals or complete the above tasks, thereby providing users Provide the desired result. It should be noted that the training device 220 may also generate corresponding preprocessing models for the target models/rules 201 corresponding to different downstream systems, such as the corresponding preprocessing models in the preprocessing module 213 and/or the preprocessing module 214.
  • the user can manually set input data (for example, text to be processed), and the manual setting can be operated through the interface provided by the I/O interface 212.
  • the client device 240 can automatically send input data (for example, text to be processed) to the I/O interface 212. If the client device 240 is required to automatically send the input data and the user's authorization is required, the user can log in to the client device Set corresponding permissions in 240. The user can view the result output by the execution device 210 on the client device 240, and the specific presentation form may be a specific manner such as display, sound, and action.
  • the client device 240 can also be used as a data collection terminal to collect the input data of the input I/O interface 212 and the output result of the output I/O interface 212 as new sample data, and store it in the database 230 as shown in the figure.
  • the I/O interface 212 directly uses the input data input to the I/O interface 212 and the output result of the output I/O interface 212 as a new sample as shown in the figure. The data is stored in the database 230.
  • FIG. 4 is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship among the devices, devices, modules, etc. shown in the figure does not constitute any limitation.
  • the data storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 may also be placed in the execution device 210.
  • the target model/rule 201 is obtained by training according to the training device 220.
  • the target model/rule 201 may be the target processing model in the embodiment of the present application.
  • the target processing model provided in the embodiment of the present application may be Neural network model.
  • it can be CNN, deep convolutional neural network (deep convolutional neural network, DCNN).
  • CNN is a very common neural network
  • the structure of CNN will be introduced in detail below in conjunction with Figure 5.
  • a convolutional neural network is a deep neural network with a convolutional structure. It is a deep learning architecture.
  • the deep learning architecture refers to the algorithm of machine learning. Multi-level learning is carried out on the abstract level of.
  • CNN is a feed-forward artificial neural network. Each neuron in the feed-forward artificial neural network can respond to the input image.
  • a convolutional neural network (CNN) 300 may include an input layer 310, a convolutional layer/pooling layer 320 (the pooling layer is optional), and a neural network layer 330.
  • CNN convolutional neural network
  • the convolutional layer/pooling layer 320 may include layers 321-326, for example: in one implementation, layer 321 is a convolutional layer, layer 322 is a pooling layer, and layer 323 is a convolutional layer. Layers, 324 is a pooling layer, 325 is a convolutional layer, and 326 is a pooling layer; in another implementation, 321 and 322 are convolutional layers, 323 is a pooling layer, and 324 and 325 are convolutional layers. Layer, 326 is the pooling layer. That is, the output of the convolutional layer can be used as the input of the subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.
  • the convolution layer 321 can include many convolution operators.
  • the convolution operator is also called a kernel. Its role in natural language processing is equivalent to a filter that extracts specific information from the input speech or semantic information.
  • the operator can essentially be a weight matrix, which is usually predefined.
  • weight values in these weight matrices need to be obtained through a lot of training in practical applications.
  • Each weight matrix formed by the weight values obtained through training can extract information from the input data, thereby helping the convolutional neural network 300 to make correct predictions.
  • the initial convolutional layer (such as 321) often extracts more general features, which can also be called low-level features; with the convolutional neural network
  • the features extracted by the subsequent convolutional layers (for example, 326) become more and more complex, such as features such as high-level semantics, and features with higher semantics are more suitable for the problem to be solved.
  • pooling layer after the convolutional layer, that is, the 321-326 layers as illustrated by 320 in Figure 5, which can be a convolutional layer followed by a layer
  • the pooling layer can also be a multi-layer convolutional layer followed by one or more pooling layers.
  • the sole purpose of the pooling layer is to reduce the size of the data space.
  • Neural network layer 330
  • the convolutional neural network 300 After processing by the convolutional layer/pooling layer 320, the convolutional neural network 300 is not enough to output the required output information. Because as mentioned above, the convolutional layer/pooling layer 320 only extracts features and reduces the parameters brought by the input data. However, in order to generate the final output information (the required class information or other related information), the convolutional neural network 300 needs to use the neural network layer 330 to generate one or a group of required classes of output. Therefore, the neural network layer 330 may include multiple hidden layers (331, 332 to 33n as shown in FIG. 5) and an output layer 340. The parameters contained in the hidden layers may be based on specific task types. Relevant training data of is obtained through pre-training. For example, the task type may include speech or semantic recognition, classification or generation, and so on.
  • the output layer 340 has a loss function similar to the classification cross entropy, which is specifically used to calculate the prediction error.
  • the convolutional neural network 300 shown in FIG. 5 is only used as an example of a convolutional neural network. In specific applications, the convolutional neural network may also exist in the form of other network models.
  • a convolutional neural network (CNN) 300 may include an input layer 310, a convolutional layer/pooling layer 320 (the pooling layer is optional), and a neural network layer 330.
  • CNN convolutional neural network
  • Multiple convolutional layers/pooling layers in the convolutional layer/pooling layer 320 are in parallel, and the respectively extracted features are input to the full neural network layer 330 for processing.
  • FIG. 7 is a schematic diagram of the hardware structure of a chip provided by an embodiment of the application.
  • the chip includes a neural network processor (neural processing unit, NPU) 40.
  • the chip can be set in the execution device 110 as shown in FIG. 4 to complete the calculation work of the calculation module 111.
  • the chip can also be set in the training device 120 as shown in FIG. 4 to complete the training work of the training device 120 and output the target model/rule 101.
  • the algorithms of each layer in the convolutional neural network as shown in FIG. 5 and FIG. 6 can all be implemented in the chip as shown in FIG. 7.
  • the NPU 40 can be mounted on a host CPU, and the host CPU distributes tasks.
  • the core part of the NPU 40 is the arithmetic circuit 403.
  • the controller 404 in the NPU 40 can control the arithmetic circuit 403 to extract data from the memory (weight memory or input memory) and perform calculations.
  • the arithmetic circuit 403 includes multiple processing units (process engines, PE). In some implementations, the arithmetic circuit 403 is a two-dimensional systolic array. The arithmetic circuit 403 may also be a one-dimensional systolic array or other electronic circuit capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 403 is a general-purpose matrix processor.
  • the arithmetic circuit fetches the data corresponding to matrix B from the weight memory 402 and caches it on each PE in the arithmetic circuit.
  • the arithmetic circuit fetches the matrix A data and matrix B from the input memory 401 to perform matrix operations, and the partial result or final result of the obtained matrix is stored in an accumulator 408.
  • the vector calculation unit 407 can perform further processing on the output of the arithmetic circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, and so on.
  • the vector calculation unit 407 can be used for network calculations in the non-convolutional/fully connected layers (FC) layers of the neural network, such as pooling, batch normalization, and partial response. Normalization (local response normalization), etc.
  • FC non-convolutional/fully connected layers
  • the vector calculation unit 407 can store the processed output vector to the unified buffer 406.
  • the vector calculation unit 407 may apply a nonlinear function to the output of the arithmetic circuit 403, such as a vector of accumulated values, to generate the activation value.
  • the vector calculation unit 407 generates a normalized value, a combined value, or both.
  • the processed output vector can be used as an activation input to the arithmetic circuit 403, for example for use in subsequent layers in a neural network.
  • the unified memory 406 is used to store input data and output data.
  • the weight data directly transfers the input data in the external memory to the input memory 401 and/or the unified memory 406 through the storage unit access controller 405 (direct memory access controller, DMAC), and stores the weight data in the external memory into the weight memory 402, And the data in the unified memory 406 is stored in the external memory.
  • DMAC direct memory access controller
  • the bus interface unit (BIU) 410 is used to implement interaction between the main CPU, the DMAC, and the fetch memory 409 through the bus.
  • An instruction fetch buffer 409 connected to the controller 404 is used to store instructions used by the controller 404;
  • the controller 404 is used to call the instructions cached in the memory 409 to control the working process of the computing accelerator.
  • the unified memory 406, the input memory 401, the weight memory 402, and the fetch memory 409 may all be on-chip memories.
  • the external memory of the NPU may be a memory external to the NPU, and the external memory may be a double data rate synchronous dynamic random access memory (double data rate synchronous dynamic random access memory, DDR SDRAM), high bandwidth memory (high bandwidth memory, HBM), or Other readable and writable memory.
  • DDR SDRAM double data rate synchronous dynamic random access memory
  • HBM high bandwidth memory
  • Other readable and writable memory it should be understood that the chip hardware result shown in FIG. 7 is only an exemplary illustration, and the application is not limited thereto.
  • FIG. 8 is a schematic diagram of a system structure in a translation scenario provided by an embodiment of the application. As shown in FIG. 8, the text processing method in the embodiment of the present application may be executed by a natural language understanding (NLU) cloud-side module.
  • NLU natural language understanding
  • the system includes vision module, OCR engine module, OCR recognition module, NLU module, NLU cloud side module, translation module and translation cloud module.
  • Vision module used to collect pictures.
  • the vision module can collect pictures by taking pictures.
  • OCR engine module used for scheduling of OCR tasks.
  • OCR recognition module used to realize character recognition based on OCR algorithm.
  • NLU module used for scheduling of NLU-related tasks.
  • NLU cloud side module used to correct wrong words/grammar in the received text.
  • Translation module used for scheduling translation tasks among multiple languages.
  • Translation cloud module used to translate the received text.
  • the vision module transmits the collected pictures to the OCR engine module.
  • the OCR engine module transmits the picture to the OCR recognition module through scheduling.
  • the OCR recognition module recognizes the text in the picture, that is, the original text, and returns the original text to the OCR engine module.
  • the OCR engine module transmits the original text to the NLU module.
  • the NLU module transmits the original text to the NLU cloud-side module through scheduling.
  • S6 The NLU cloud-side module corrects the wrong words/grammar in the original text to obtain the corrected original text.
  • the OCR engine module transmits the corrected original text to the translation module.
  • the translation module transmits the corrected original text to the translation cloud module through scheduling.
  • the translation cloud module performs translation, obtains the translation, and sends it back to the translation module.
  • the translation module returns the translation to the OCR engine module.
  • the text processing method is used for text error correction, that is, the text processing model can be a text error correction model. Input the text to be processed into the text processing model for error correction processing, and then the correction result of the text to be processed can be obtained.
  • FIG. 8 is only an example of the text processing method in the embodiment of the present application.
  • the text processing model may be a text translation model, and the text to be processed is input into the text translation model for correction. Error processing, and the result of error correction is translated to obtain the translated text of the text to be processed.
  • the text processing model in Figure 8 is deployed on a cloud server. It should be understood that the text processing model can also be deployed on smart terminal devices.
  • the smart terminal may be an electronic device with a camera.
  • the smart terminal may be a mobile phone with image processing function, a tablet personal computer (TPC), a media player, a smart TV, a laptop computer (LC) , A personal digital assistant (PDA), a personal computer (PC), or a vehicle-mounted terminal in an autonomous vehicle, etc., which are not limited in the embodiment of the present application.
  • FIG. 9 is a schematic flowchart of a text processing method provided by an embodiment of the present application.
  • the text processing method shown in FIG. 9 can be executed by a text processing device, which can be the data processing device in FIG. 1, or the user equipment in FIG. 2, or the execution device 110 or the local device in FIG. , It may also be the execution device 210 in FIG. 4.
  • the method shown in FIG. 8 includes steps 510 to 550, and steps 510 to 550 are respectively described in detail below.
  • the text to be processed may be OCR output text, or may be text input by the user.
  • the text input by the user may include content posted in a social network, or may be content entered in a search box of a search engine, and so on.
  • the text to be processed may be any text that requires error correction, and the embodiment of the present application does not limit the specific form of the text to be processed.
  • Errors in the text to be processed may include non-word errors and real-word errors.
  • Non-word errors mean that the words in the text to be processed are not in the preset vocabulary.
  • True word error means that the words in the text to be processed exist in the preset lexicon, but cause problems with context and semantics, and are not words required by the current context.
  • wrong words can include non-words and wrong true words.
  • non-word error detection may be performed on the text to be processed based on the first preset vocabulary to obtain the non-word in the text to be processed.
  • the first preset vocabulary can be used to distinguish true words from non-words.
  • True words refer to words that exist in the first preset thesaurus, and correspondingly, non-words refer to words that do not exist in the first preset thesaurus.
  • the vocabulary that can be used to detect non-word errors can all be understood as the first preset vocabulary.
  • the first preset thesaurus may be an English dictionary.
  • Non-words are words that do not exist in English dictionaries, for example, wasld.
  • the embodiment of the present application does not limit the type of the first preset word database.
  • the "thesaurus” may also be referred to as a "dictionary” or a “vocabulary”.
  • the true word error detection of the text to be processed may be performed based on the language model to obtain the false true word in the text to be processed. For example, when the perplexity of the text corresponding to a word is higher than a set threshold, the word is judged to be a false true word.
  • the language model may be a statistical language model, for example, an n-gram model.
  • the statistical language model is more dominant in the extraction of semantic information of short and medium texts, and is suitable for scenarios that rely less on long-distance semantic information, such as text error correction in OCR scenarios.
  • the language model may also be a neural network model, for example, a recurrent neural network (recurrent neural network, RNN) model.
  • a recurrent neural network recurrent neural network, RNN
  • the wrong word in step 520 may include only non-words, or only wrong true words, and may also include non-words and wrong true words.
  • the wrong word candidate word library is used to indicate the candidate word set corresponding to multiple wrong words.
  • a wrong word can correspond to one or more candidate words.
  • a candidate word can also correspond to one or more wrong words.
  • the candidate word set corresponding to a wrong word may include only one candidate word.
  • the wrong word candidate word database may be a wrong word candidate word database generated offline.
  • the wrong word includes the wrong true word
  • the wrong word candidate word database may include the wrong true word candidate word database.
  • step 530 may include step 531.
  • the false true word candidate word library includes a plurality of false true words corresponding to candidate word sets. There can be one or more candidate words corresponding to a false true word.
  • the lexicon of false true word candidates can be generated offline.
  • the candidate word set corresponding to the multiple false true words may be determined based on the similarity between the multiple false true words and the candidate words corresponding to the multiple false true words.
  • the similarity between the multiple false true words and the candidate words corresponding to the multiple false true words may include the edit distance and/or commonality between the multiple false true words and the candidate words corresponding to the multiple false true words. String. That is, the candidate word corresponding to the wrong true word can be determined based on the minimum edit distance and/or the maximum common character string.
  • the minimum edit distance refers to the minimum number of editing operations required to convert one word to another. Editing operations include operations such as insertion, deletion, translocation, and replacement of characters in a word.
  • the largest common character string refers to the number of consecutive identical characters contained in two words.
  • similarity may also be other forms of similarity, such as character similarity.
  • one false true word in the false true word candidate word database is word
  • the candidate words corresponding to the false true word may include world, words, and sword.
  • the candidate word corresponding to the word can be determined as at least one of world, words, and sword according to the false true word candidate word library, as the candidate word set of the word.
  • the wrong word may include a non-word
  • the wrong word candidate word database may include a non-word candidate word database.
  • step 530 may include:
  • the non-word candidate word vocabulary may include multiple candidate word sets corresponding to the non-words.
  • one non-word in the non-word candidate word library is wasld
  • the candidate words corresponding to wereld may include world, word, and sword.
  • the candidate word corresponding to wasld can be determined as at least one of world, word, and sword according to the non-word candidate word library, as the candidate word set of wasld.
  • the candidate word sets corresponding to multiple non-words in the non-word candidate word library may be determined based on the similarity between the multiple non-words and the candidate words corresponding to the multiple non-words. There can be one or more candidate words corresponding to a non-word. In other words, there can be only one candidate word in the candidate word set corresponding to a non-word.
  • the similarity between multiple non-words and candidate words corresponding to multiple non-words may include edit distances and/or common character strings between multiple non-words and candidate words corresponding to multiple non-words. That is, the candidate words corresponding to the non-words can be determined based on the minimum edit distance and/or the maximum common character string.
  • similarity may also be other forms of similarity, such as character similarity.
  • non-words For non-words, the number of non-words is very large. For example, word corresponds to 456976 (26 to the 4th power) non-words. If a non-word vocabulary is established for a 700,000-word English dictionary, the number of non-words will reach tens of billions, and the storage and error checking will be extremely expensive.
  • the following gives another implementation way of generating candidate words corresponding to non-words based on a pre-stored non-word candidate word library.
  • the non-word candidate word library includes candidate word sets corresponding to multiple common subwords. There can be one or more candidate words corresponding to a common subword. That is to say, the candidate word set of a common subword can include only one candidate word.
  • step 532 may include step 532a and step 532b.
  • 532a Generate common subwords corresponding to non-words in the text to be processed.
  • the similarity between the common subword and the non-word satisfies the preset condition.
  • the similarity between the common subword and the non-word may include an edit distance between the common subword and the non-word.
  • the preset condition may be that the maximum edit distance between the common subword and the non-word is less than a preset value.
  • the preset value may be 2.
  • the maximum edit distance between the non-word and the common sub-word corresponding to the non-word is limited to 2, and the common sub-word corresponding to the non-word in the text to be processed is generated. That is, the number of operations in the process of generating the common subword corresponding to the non-word from the non-word does not exceed two times.
  • the way of generating the common subword corresponding to the non-word may be determined by the probability of the OCR identifying a letter incorrectly. For example, OCR has a higher probability of recognizing o as e. For the non-word wasd, the letter e can be deleted to obtain the common subword wrd corresponding to wered.
  • 532b Determine, according to the non-word candidate word library, a candidate word set corresponding to a common subword corresponding to a non-word in the text to be processed as a candidate word set corresponding to the non-word in the text to be processed.
  • the common subword is equivalent to the bridge between the candidate word and the non-word, and the candidate word corresponding to the non-word can be generated indirectly through the common subword.
  • the non-word candidate word database can be generated offline.
  • the candidate word set corresponding to multiple common subwords in the non-word candidate word library may be determined based on the similarity between the multiple common subwords and the candidate words corresponding to the multiple common subwords.
  • the similarity between multiple common subwords and multiple candidate words corresponding to non-words may include edit distances and/or common characters between the multiple common subwords and candidate words corresponding to the multiple common subwords string. That is, the candidate words corresponding to the common subword can be determined based on the minimum edit distance and/or the maximum common character string.
  • similarity may also be other forms of similarity, such as character similarity.
  • the common subwords of the word are first generated online, and the number of common subwords is only n+n*(n-1)/2, and then the candidate words of the word are determined according to the candidate words corresponding to the common subwords. Obviously, by generating the common sub-words of the word online, and then determining the candidate words of the non-word, the time consumption of text error correction can be significantly reduced.
  • the non-word in step 530 may be a non-word after preprocessing.
  • This preprocessing can include filtering out words in a special format.
  • Specially formatted words can be words that meet preset criteria.
  • words in a special format can include combined words.
  • a combined word refers to a word formed by at least two words due to lack of spaces. For example, inChina.
  • the preprocessing method is not limited in the embodiments of the present application.
  • determining the target candidate word corresponding to the wrong word in the candidate word set corresponding to the wrong word may be randomly determining the target candidate word corresponding to the wrong word among the candidate words corresponding to the wrong word.
  • step 540 may include step 541 and step 542.
  • the perplexity of the candidate words in the candidate word set corresponding to the wrong word is used to indicate the possibility of the candidate words in the candidate word set corresponding to the wrong word appearing in the text to be processed.
  • the similarity between the candidate words in the candidate word set corresponding to the wrong word and the wrong word may include: the morphological similarity between the candidate word and the wrong word in the candidate word set corresponding to the wrong word and the candidates in the candidate word set corresponding to the wrong word The edit distance between the word and the wrong word.
  • Morphological similarity is used to measure the similarity of two words in morphological characteristics. For example, based on the maximum common character string and character similarity, the morphological similarity of the candidate words in the candidate word set corresponding to the wrong word and the wrong word can be judged, that is, the score corresponding to the morphological similarity can be obtained.
  • the perplexity of the candidate words in the candidate word set corresponding to the wrong word can be scored by the language model.
  • the score corresponding to each candidate word can be obtained by weighting the scores corresponding to the above items, that is, the weights are set for the scores corresponding to each item.
  • the weight can be preset or obtained through training.
  • Scoring is based on the similarity and perplexity of the text, and the similarity between the wrong word and the candidate word and the semantic information of the text to be processed are taken into consideration, so that a better candidate word can be obtained, and a more accurate scoring result can be obtained.
  • the wrong words may include non-words and wrong true words.
  • different methods can be used to determine target candidate words, and the same method can also be used to determine target candidate words.
  • step 541 and step 542 can be performed to obtain candidate words corresponding to the non-words in the text to be processed.
  • the target candidate words corresponding to the false true words in the text to be processed are randomly determined from the candidate words corresponding to the false true words in the text to be processed.
  • Correcting the wrong word may include replacing the wrong word with the target candidate word corresponding to the wrong word, and also including not processing the wrong word, that is, not replacing the wrong word.
  • step 550 may be step 550a.
  • step 550 may be step 550b.
  • the wrong word is not corrected. That is, the target candidate word corresponding to the wrong word is not used to replace the wrong word. This can reduce time consumption and quickly implement text error correction.
  • step 541 when the morphological similarity is lower than the preset threshold, the candidate word with the second highest score in step 541 can be used as the target candidate word corresponding to the wrong word, and step 550b is repeated until the morphological similarity meets the preset Threshold target candidate words, and use target candidate words to replace wrong words.
  • the text output by OCR should have high morphological similarity with the correct text.
  • the morphological similarity of the target candidate words corresponding to the wrong word and the wrong word is lower than a certain threshold, it is the morphology of the two The similarity is low. In this case, the wrong words are not corrected to avoid introducing new errors.
  • step 550 may be step 550c.
  • 550c Detect the perplexity of the text containing the target candidate word corresponding to the wrong word through the language model. When the perplexity is lower than the perplexity threshold, replace the wrong word with the target candidate word corresponding to the wrong word as the correction result of the wrong word .
  • the wrong word is not corrected. That is, the target candidate word corresponding to the wrong word is not used to replace the wrong word. This can reduce time consumption and quickly implement text error correction.
  • the candidate word with the second highest score in step 541 can be used as the target candidate word corresponding to the wrong word, and step 550c is repeated until the target candidate whose perplexity meets the predetermined threshold is obtained. Word, and replace the wrong word with the target candidate word.
  • step 550 different steps may be performed for the non-words in the text to be processed and the wrong true words in the text to be processed, or the same steps may be performed.
  • step 550a can be executed, and for false true words in the text to be processed, step 550b or 550a can be executed.
  • step 550 can be performed separately, or step 550 can be performed together.
  • candidate words corresponding to the wrong word can be quickly generated, with a small amount of calculation, reducing the time consumption of text error correction, and ensuring the real-time nature of text processing.
  • a variety of methods are used to score candidate words to obtain better candidate words, which improves the accuracy of text error correction.
  • the target candidate word with the morphological similarity higher than or equal to the preset threshold is used to correct the wrong word, which can avoid the introduction of new errors.
  • FIG. 10 is a schematic flowchart of a text processing method 600 provided by an embodiment of the present application.
  • the method 600 may be a specific example of the method 500.
  • the method 600 includes steps 610 to 6140. Steps 610 to 6140 will be described in detail below.
  • method 600 may further include step 611.
  • the length of the text to be processed refers to the number of words in the text to be processed.
  • the preset length can be 2.
  • the English vocabulary is an example of the first preset vocabulary in the method 500.
  • Step 620 is used to obtain non-words and true words in the text to be processed.
  • Non-words are words that do not exist in the English thesaurus.
  • True words are words that exist in the English thesaurus.
  • Step 630 is performed for non-words in the text to be processed. Perform 690 for the true words in the text to be processed.
  • Non-word 1# can include one non-word or multiple non-words.
  • Public subword 1# refers to the public subword corresponding to non-word 1#.
  • the common subword 1# can be one or more.
  • the similarity between the common subword 1# and the non-word 1# may include the edit distance between the common subword 1# and the non-word 1#.
  • the preset condition may be that the maximum edit distance between the common subword 1# and the non-word 1# is less than the preset value.
  • the preset value may be 2.
  • the maximum edit distance between the non-word 1# and the common subword 1# in the text to be processed is limited to 2, and the common subword 1# is generated.
  • the common subword candidate word database is an example of the non-word candidate word database in the method 500.
  • the common subword candidate word library includes candidate word sets corresponding to multiple common subwords.
  • the candidate word set corresponding to the common subword 1# is determined as the candidate word set corresponding to the non-word 1# according to the common subword candidate word library.
  • the candidate words in the candidate word set corresponding to non-word 1# can be scored through language model, edit distance, and maximum common character string.
  • the detailed process is as described in step 541 in method 500, which will not be repeated here.
  • the candidate word with the highest score in step 660 may be used as the target candidate word corresponding to non-word 1#.
  • the target candidate word corresponding to the non-word 1# is used to replace the non-word 1#, that is, the non-word 1#
  • the corresponding target candidate word is regarded as the correction result of non-word 1#.
  • the non-word 1# is taken as the correction result of the non-word 1#, that is, the non-word 1 is not correct. #Processing.
  • the word is judged to be a false true word.
  • Step 6100 is executed for the wrong true words in the text to be processed.
  • the false true word in the text to be processed is called false true word 1#.
  • the false true word 1# can include one false true word or multiple false true words.
  • the false true word candidate word library includes a plurality of false true words corresponding to candidate word sets.
  • the candidate word set corresponding to the false true word 1# is determined according to the false true word candidate word library.
  • the candidate words in the candidate word set corresponding to the wrong true word 1# can be scored through the language model, the edit distance, and the largest common character string.
  • the detailed process is as described in step 541 in method 500, which will not be repeated here.
  • the candidate word with the highest score in step 6120 may be used as the target candidate word corresponding to the false true word 1#.
  • 6140 Determine the morphological similarity between the target candidate word corresponding to the wrong true word 1# and the wrong true word 1#. If the morphological similarity between the target candidate word corresponding to erroneous true word 1# and erroneous true word 1# is higher than or equal to the preset threshold, the target candidate word corresponding to erroneous true word 1# is used to replace erroneous true word 1#, That is, the target candidate word corresponding to the wrong true word 1# is used as the correction result of the wrong true word 1#.
  • the wrong true word 1# is taken as the correction result of the wrong true word 1#, That is, the wrong true word 1# is not processed.
  • the candidate word set corresponding to the false true word and the candidate word set corresponding to the non-word can be quickly generated, and the amount of calculation is relatively large. Small, reduce the time consumption of text error correction, and ensure the real-time nature of text processing.
  • using multiple methods to score candidate words can obtain better candidate words and improve the accuracy of text error correction.
  • the target candidate word with the morphological similarity higher than or equal to the preset threshold is used to correct the wrong word, which can avoid the introduction of new errors. Originally recognized correct words were modified as wrong words.
  • FIG. 11 is a schematic block diagram of a text processing device provided by an embodiment of the present application. It should be understood that the text processing apparatus 1000 can execute the text processing method shown in FIG. 9 or FIG. 10.
  • the text processing device 1000 includes: an acquiring unit 1010 and a processing unit 1020.
  • the obtaining unit 1010 is used to obtain the text to be processed; the processing unit 1020 is used to: perform error detection processing on the text to be processed to obtain the wrong words in the text to be processed; according to the pre-stored wrong word candidate word database Determine the candidate word set corresponding to the wrong word, the wrong word candidate word library is used to indicate the candidate word set corresponding to multiple wrong words; determine the target candidate word corresponding to the wrong word in the candidate word set corresponding to the wrong word; The target candidate word corrects the wrong word.
  • the processing unit 1020 is configured to: determine the candidate word corresponding to the wrong word according to the similarity between the candidate word in the candidate word set corresponding to the wrong word and the wrong word and the perplexity of the candidate word in the candidate word set corresponding to the wrong word
  • the candidate words in the set are scored, where the perplexity of the candidate words in the candidate word set corresponding to the wrong word is used to indicate the possibility of the candidate words in the candidate word set corresponding to the wrong word appearing in the text to be processed;
  • the candidate word with the highest score in the candidate word set is determined as the target candidate word corresponding to the wrong word.
  • the similarity between the candidate word in the candidate word set corresponding to the wrong word and the wrong word includes the morphological similarity between the candidate word and the wrong word in the candidate word set corresponding to the wrong word and the candidate word set corresponding to the wrong word Edit distance between candidate words and wrong words
  • the wrong word includes a non-word
  • the wrong word candidate word database includes a non-word candidate word database
  • the non-word candidate word library includes candidate word sets corresponding to multiple common subwords
  • the processing unit 1020 is configured to: generate common subwords corresponding to the non-words, wherein the common subwords and non-words are similar The gender meets the preset condition; the candidate word set corresponding to the common subword is determined as the candidate word set corresponding to the non-word according to the non-word candidate word library.
  • the candidate word set corresponding to multiple common subwords in the non-word candidate word library is determined based on the similarity between the multiple common subwords and the candidate words corresponding to the multiple common subwords.
  • the wrong word includes a wrong true word
  • the wrong word candidate word library includes a wrong true word candidate word library
  • the wrong true word candidate word library includes a plurality of candidate word sets corresponding to the wrong true words.
  • the candidate word set corresponding to multiple false true words in the false true word candidate word database is determined based on the similarity between the multiple false true words and the candidate words corresponding to the multiple false true words.
  • the processing unit 1020 is configured to: determine the morphological similarity between the target candidate word corresponding to the wrong word and the wrong word; when the morphological similarity is higher than or equal to a preset threshold, the target candidate corresponding to the wrong word The word is the result of the correction of the wrong word.
  • a "unit” can be a software program, a hardware circuit, or a combination of the two that realize the above-mentioned functions.
  • the hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, and a processor for executing one or more software or firmware programs (such as a shared processor, a dedicated processor, or a group processor). Etc.) and memory, merged logic circuits and/or other suitable components that support the described functions.
  • the units of the examples described in the embodiments of the present application can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.
  • FIG. 12 is a schematic diagram of the hardware structure of a text processing device provided by an embodiment of the present application.
  • the text processing apparatus 1200 shown in FIG. 12 includes a memory 1201, a processor 1202, a communication interface 1203, and a bus 1204.
  • the memory 1201, the processor 1202, and the communication interface 1203 implement communication connections between each other through the bus 1204.
  • the memory 1201 may be a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM).
  • the memory 1201 may store a program.
  • the processor 1202 is configured to execute each step of the text processing method of the embodiment of the present application, for example, execute each step shown in FIG. 9 or FIG. step.
  • the text processing apparatus shown in the embodiment of the present application may be a smart terminal or a chip configured in the smart terminal.
  • the text processing method disclosed in the foregoing embodiments of the present application may be applied to the processor 1202 or implemented by the processor 1202.
  • the processor 1202 may be an integrated circuit chip with signal processing capabilities.
  • the steps of the above-mentioned text processing method can be completed by an integrated logic circuit of hardware in the processor 1202 or instructions in the form of software.
  • the processor 1202 may be a chip including the NPU shown in FIG. 7.
  • the aforementioned processor 1202 may be a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose processor, a digital signal processor (DSP), and an application specific integrated circuit (application integrated circuit).
  • CPU central processing unit
  • GPU graphics processing unit
  • DSP digital signal processor
  • ASIC application specific integrated circuit
  • ASIC application specific integrated circuit
  • FPGA field programmable gate array
  • the general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like.
  • the steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor.
  • the software module can be located in random access memory (RAM), flash memory, read-only memory (read-only memory, ROM), programmable read-only memory or electrically erasable programmable memory, registers, etc. mature in the field Storage medium.
  • the storage medium is located in the memory 1201, and the processor 1202 reads the information in the memory 1201, and combines its hardware to complete the functions required by the units included in the text processing device shown in FIG. 11 in the implementation of this application, or execute the method of this application The text processing method shown in FIG. 9 or FIG. 10 of the embodiment.
  • the communication interface 1203 uses a transceiver device such as but not limited to a transceiver to implement communication between the device 1200 and other devices or a communication network.
  • a transceiver device such as but not limited to a transceiver to implement communication between the device 1200 and other devices or a communication network.
  • the bus 1204 may include a path for transferring information between various components of the text processing apparatus 1200 (for example, the memory 1201, the processor 1202, and the communication interface 1203).
  • the text processing apparatus 1200 only shows a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art should understand that the text processing apparatus 1200 may also include other necessary for normal operation. Device. At the same time, according to specific needs, those skilled in the art should understand that the above-mentioned text processing apparatus 1200 may also include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the above-mentioned text processing apparatus 1200 may also only include the necessary components to implement the embodiments of the present application, and not necessarily include all the components shown in FIG. 12.
  • the embodiment of the present application also provides a chip, which includes a transceiver unit and a processing unit.
  • the transceiver unit may be an input/output circuit or a communication interface;
  • the processing unit is a processor, microprocessor, or integrated circuit integrated on the chip.
  • the chip can execute the method in the above method embodiment.
  • the embodiment of the present application also provides a computer-readable storage medium on which an instruction is stored, and the method in the foregoing method embodiment is executed when the instruction is executed.
  • the embodiments of the present application also provide a computer program product containing instructions, which execute the methods in the above method embodiments when the instructions are executed.
  • the memory may include a read-only memory and a random access memory, and provide instructions and data to the processor.
  • a part of the processor may also include a non-volatile random access memory.
  • the processor may also store device type information.
  • the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not correspond to the embodiments of the present application.
  • the implementation process constitutes any limitation.
  • the disclosed system, device, and method can be implemented in other ways.
  • the device embodiments described above are merely illustrative, for example, the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components may be combined or It can be integrated into another system, or some features can be ignored or not implemented.
  • the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.
  • the functional modules in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.
  • the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium.
  • the technical solution of the present application essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disks or optical disks and other media that can store program codes. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Artificial Intelligence (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Evolutionary Computation (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Databases & Information Systems (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Machine Translation (AREA)

Abstract

人工智能领域中自然语言处理领域的文本处理方法以及装置，该方法包括：获取待处理文本（510）；对待处理文本进行检错处理，得到待处理文本中的错词（520）；根据预先存储的错词候选词词库确定错词对应的候选词集（530）；在错词对应的候选词集中确定错词对应的目标候选词（540）；根据错词对应的目标候选词对错词进行校正（550）。该技术方案能够提高候选词的生成速度，降低了文本纠错的时间消耗。

Description

文本处理方法及装置
本申请要求于2019年12月23日提交中国专利局、申请号为201911335094.7、申请名称为“文本处理方法及装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及自然语言处理领域，并且更具体地，涉及一种文本处理方法及装置。
背景技术
人工智能(artificial intelligence，AI)是利用数字计算机或者数字计算机控制的机器模拟、延伸和扩展人的智能，感知环境、获取知识并使用知识获得最佳结果的理论、方法、技术及应用系统。换句话说，人工智能是计算机科学的一个分支，它企图了解智能的实质，并生产出一种新的能以人类智能相似的方式作出反应的智能机器。人工智能也就是研究各种智能机器的设计原理与实现方法，使机器具有感知、推理与决策的功能。
随着人工智能技术的不断发展，让人机之间能够通过自然语言进行交互的自然语言人机交互系统变的越来越重要。人机之间能够通过自然语言进行交互，就需要系统能够识别出人类自然语言的具体含义。通常，系统通过采用对自然语言的句子进行关键信息提取来识别句子的具体含义。
文本纠错(text error correction)是对原始文本进行错误检测(error detection)，并根据自然语言处理技术对错误进行纠正。目前，通常有两种方法可以实现文本纠错。一种是基于字典判断输入的查询词是否正确，检测出错词，并生成错词对应的候选词，利用候选词对错词进行纠正。另一种是通过语言模型对上下文语义信息进行提取，检测出错词，并生成错词对应的候选词，利用候选词对错词进行纠正。上述方法均耗时较长，对设备的计算能力或者通讯时延的要求较高，很大程度上影响了文本纠错端到端的快速实现。
发明内容
本申请提供一种文本处理方法及装置，能够提高候选词的生成速度，降低了文本纠错的时间消耗。
第一方面，提供了一种文本处理方法，包括：获取待处理文本；对待处理文本进行检错处理，得到待处理文本中的错词；根据预先存储的错词候选词词库确定错词对应的候选词集，错词候选词词库用于指示多个错词对应的候选词集；在错词对应的候选词集中确定错词对应的目标候选词；根据错词对应的目标候选词对错词进行校正。
待处理文本可以是光学字符识别(optical character recognition，OCR)输出文本，或者，可以是用户输入的文本。例如，用户输入的文本可以包括社交网络中发布的内容，或者可以是搜索引擎的搜索框中输入的内容等。应理解，待处理文本可以是任意需要进行纠 错的文本，本申请对待处理文本的具体形式不做限定。
可选地，错词可以包括非词。具体地，可以基于第一预设词库对待处理文本进行非词错误检测，得到待处理文本中的非词。非词指的是不存在于第一预设词库中的词。
应理解，能够用于检测非词错误的词库均可以理解为第一预设词库。例如，在方法500应用于英文文本纠错的情况下，第一预设词库可以为英文字典。非词即为不存在于英文字典的词，例如，werld。
可选地，错词可以包括错误真词。具体地，可以基于语言模型对待处理文本进行真词错误检测，得到待处理文本中的错误真词。
例如，该语言模型可以为统计语言模型。再如，该语言模型也可以为神经网络模型。
应理解，错词可以仅包括非词，也可以仅包括错误真词，还可以包括非词和错误真词。检错处理可以包括检测非词错误，也可以包括检测真词错误，还可以既包括检测非词错误，也包括检测真词错误。
根据预先存储的错词候选词词库生成错词对应的候选词集中的候选词可以为一个，也可以为多个。
该错词候选词词库可以是离线生成的错词候选词词库。
在本申请的实施例中，根据预设的错词候选词词库，能够快速生成错词对应的候选词，计算量较小，降低了文本纠错的时间消耗，保证了文本处理的实时性。
结合第一方面，在一种可能的实现方式中，在错词对应的候选词集中确定错词对应的目标候选词，包括：根据错词对应的候选词集中的候选词与错词之间的相似性以及错词对应的候选词集中的候选词的困惑度对错词对应的候选词集中的候选词进行评分，其中，错词对应的候选词集中的候选词的困惑度用于指示错词对应的候选词集中的候选词在待处理文本中出现的可能性；将错词对应的候选词集中评分最高的候选词确定为错词对应的目标候选词。
错词对应的候选词集中的候选词的困惑度可以通过语言模型进行评分。
每个候选词对应的评分可以由上述几项对应的评分进行加权得到，也就是为每一项对应的评分设置权重。该权重可以是预先设定的，也可以是训练得到的。
在本申请的实施例中，采用候选词与错词之间的相似性以及候选词的困惑度对候选词进行评分，同时考虑了错词与候选词的相似性以及待处理文本的语义信息，能够得到较优的候选词，提高了文本纠错的准确性。
结合第一方面，在一种可能的实现方式中，错词对应的候选词集中的候选词与错词之间的相似性包括错词对应的候选词集中的候选词与错词之间的形态相似性和错词对应的候选词集中的候选词与错词之间的编辑距离。
形态相似性用于衡量两个词在形态特征上的相似性。例如，可以基于最大公共字符串和字符相似性对错词和错词对应的候选词集中的候选词进行形态特征上相似性的判断，也就是得到形态性相似性对应的评分。
结合第一方面，在一种可能的实现方式中，错词包括非词，错词候选词词库包括非词候选词词库。
结合第一方面，在一种可能的实现方式中，非词候选词词库包括多个公共子词对应的候选词集，以及根据预先存储的错词候选词词库确定错词对应的候选词集，包括：生成非 词对应的公共子词，其中，公共子词与非词之间的相似性满足预设条件；根据非词候选词词库将公共子词对应的候选词集确定为非词对应的候选词集。
一个非词对应的公共子词可以为一个，也可以为多个。
例如，公共子词与该非词之间的相似性可以包括该公共子词与该非词之间的编辑距离。预设条件可以为该公共子词与该非词之间的最大编辑距离小于预设值。例如，该预设值可以为2。应理解，此处仅为示意，公共子词与该非词之间的相似性也可以包括其他形式的相似性，例如，公共子词与该非词之间的相似性包括公共子词与该非词之间的形态相似性。本申请对公共子词与该非词之间的相似性的确定方式不做限定。
非词候选词词库可以是离线生成的。
在本申请的实施例中，通过公共子词对应的候选词集确定非词的候选词集，一方面，相对于直接在线生成非词的候选词集，能够显著降低文本纠错的时间消耗；另一方面，能够减少存储非词候选词词库的所需的存储空间。
结合第一方面，在一种可能的实现方式中，非词候选词词库中的多个公共子词对应的候选词集是基于多个公共子词与多个公共子词对应的候选词之间的相似性确定的。
例如，多个公共子词与多个非词对应的候选词之间的相似性可以包括多个公共子词与多个公共子词对应的候选词之间的编辑距离和/或公共字符串。也就是可以基于最小编辑距离和/或最大公共字符串确定公共子词对应的候选词。
应理解，该相似性也可以是其他形式的相似性，例如字符相似性等。
结合第一方面，在一种可能的实现方式中，错词包括错误真词，错词候选词词库包括错误真词候选词词库，错误真词候选词词库包括多个错误真词对应的候选词集。
结合第一方面，在一种可能的实现方式中，错误真词候选词词库中的多个错误真词对应的候选词集是基于多个错误真词与多个错误真词对应的候选词之间的相似性确定的。
例如，多个错误真词与多个错误真词对应的候选词之间的相似性可以包括多个错误真词与多个错误真词对应的候选词之间的编辑距离和/或公共字符串。也就是可以基于最小编辑距离和/或最大公共字符串确定错误真词对应的候选词。
应理解，该相似性也可以是其他形式的相似性，例如字符相似性等。
结合第一方面，在一种可能的实现方式中，根据错词对应的目标候选词对错词进行校正，包括：判断错词对应的目标候选词和错词之间的形态相似性；在形态相似性高于或等于预设阈值的情况下，将错词对应的目标候选词作为错词的校正结果。
在本申请实施例中，通过错词对应的目标候选词与错词的形态相似性判断，利用形态相似性高于或等于预设阈值的目标候选词对错词进行纠正，能够避免引入新的错误，例如，将原本识别正确的修改为错误的词。
第二方面，提供了一种文本处理装置，包括：获取单元和处理单元。获取单元用于获取待处理文本。处理单元用于：对待处理文本进行检错处理，得到待处理文本中的错词；根据预先存储的错词候选词词库确定错词对应的候选词集，错词候选词词库用于指示多个错词对应的候选词集；在错词对应的候选词集中确定错词对应的目标候选词；根据错词对应的目标候选词对错词进行校正。
结合第二方面，在一种可能的实现方式中，处理单元用于：根据错词对应的候选词集中的候选词与错词之间的相似性以及错词对应的候选词集中的候选词的困惑度对错词对 应的候选词集中的候选词进行评分，其中，错词对应的候选词集中的候选词的困惑度用于指示错词对应的候选词集中的候选词在待处理文本中出现的可能性；将错词对应的候选词集中评分最高的候选词确定为错词对应的目标候选词。
结合第二方面，在一种可能的实现方式中，错词对应的候选词集中的候选词与错词之间的相似性包括错词对应的候选词集中的候选词与错词之间的形态相似性和错词对应的候选词集中的候选词与错词之间的编辑距离。
结合第二方面，在一种可能的实现方式中，错词包括非词，错词候选词词库包括非词候选词词库。
结合第二方面，在一种可能的实现方式中，非词候选词词库包括多个公共子词对应的候选词集，以及处理单元用于：生成非词对应的公共子词，其中，公共子词与非词之间的相似性满足预设条件；根据非词候选词词库将公共子词对应的候选词集确定为非词对应的候选词集。
结合第二方面，在一种可能的实现方式中，非词候选词词库中的多个公共子词对应的候选词集是基于多个公共子词与多个公共子词对应的候选词之间的相似性确定的。
结合第二方面，在一种可能的实现方式中，错词包括错误真词，错词候选词词库包括错误真词候选词词库，错误真词候选词词库包括多个错误真词对应的候选词集。
结合第二方面，在一种可能的实现方式中，错误真词候选词词库中的多个错误真词对应的候选词集是基于多个错误真词与多个错误真词对应的候选词之间的相似性确定的。
结合第二方面，在一种可能的实现方式中，处理单元用于：判断错词对应的目标候选词和错词之间的形态相似性；在形态相似性高于或等于预设阈值的情况下，将错词对应的目标候选词作为错词的校正结果。
应理解，在上述第一方面中对相关内容的扩展、限定、解释和说明也适用于第二方面中相同的内容。
第三方面，提供了一种文本处理装置，该装置包括：存储器，用于存储程序；处理器，用于执行所述存储器存储的程序，当所述存储器存储的程序被所述处理器执行时所述处理器用于执行上述第一方面或第一方面中的任意一种实现方式中的文本处理方法。
第四方面，提供一种计算机可读介质，该计算机可读介质存储用于设备执行的程序代码，该程序代码包括用于上述第一方面或第一方面中的任意一种实现方式中的文本处理方法。
第五方面，提供了一种计算机程序产品，所述计算机程序产品包括：计算机程序代码，当所述计算机程序代码在计算机上运行时，使得计算机执行上述第一方面或第一方面中的任意一种实现方式中的文本处理方法。
需要说明的是，上述计算机程序代码可以全部或者部分存储在第一存储介质上，其中第一存储介质可以与处理器封装在一起的，也可以与处理器单独封装，本申请实施例对此不作具体限定。
第六方面，提供一种芯片，所述芯片包括处理器与数据接口，所述处理器通过所述数据接口读取存储器上存储的指令，执行上述第一方面或第一方面中的任意一种实现方式中的文本处理方法。
可选地，作为一种实现方式，所述芯片还可以包括存储器，所述存储器中存储有指令， 所述处理器用于执行所述存储器上存储的指令，当所述指令被执行时，所述处理器用于上述第一方面或第一方面中的任意一种实现方式中的文本处理方法。
附图说明
图1是本申请实施例提供的一种自然语言处理的应用场景示意图；
图2是本申请实施例提供的另一种自然语言处理的应用场景示意图；
图3是本申请实施例提供的自然语言处理的相关设备的示意图；
图4是本申请实施例提供的一种系统架构的示意图；
图5是本申请实施例提供的一种根据CNN模型进行文本处理的示意图；
图6本申请实施例提供的另一种根据CNN模型进行文本处理的示意图；
图7是本申请实施例提供的一种芯片的硬件结构的示意图；
图8是本申请实施例提供的一种应用场景示意图；
图9是本申请实施例提供的文本处理方法的示意性流程图；
图10是本申请实施例提供的另一种文本处理方法的示意性流程图；
图11是本申请实施例提供的文本处理装置的示意性框图；
图12是本申请实施例提供的文本处理装置的示意性框图。
具体实施方式
下面将结合附图，对本申请中的技术方案进行描述。
为了更好地理解本申请实施例的方案，下面先结合图1至图3对本申请实施例可能的应用场景进行简单的介绍。
图1示出了一种自然语言处理系统，该自然语言处理系统包括用户设备以及数据处理设备。其中，用户设备包括手机、个人电脑或者信息处理中心等智能终端。用户设备为自然语言数据处理的发起端，作为语言问答或者查询等请求的发起方，通常用户通过用户设备发起请求。
上述数据处理设备可以是云服务器、网络服务器、应用服务器以及管理服务器等具有数据处理功能的设备或服务器。数据处理设备通过交互接口接收来自智能终端的查询语句/语音/文本等问句，再通过存储数据的存储器以及数据处理的处理器环节进行机器学习，深度学习，搜索，推理，决策等方式的语言数据处理。数据处理设备中的存储器可以是一个统称，包括本地存储以及存储历史数据的数据库，数据库可以再数据处理设备上，也可以在其它网络服务器上。
在图1所示的自然语言处理系统中，用户设备可以接收用户的指令，例如，用户设备可以接收用户输入的一段文本，然后向数据处理设备发起请求，使得数据处理设备针对用户设备得到的该一段文本执行自然语言处理应用(例如，文本分类、文本序列标注、翻译等)，从而得到针对该一段文本的对应的自然语言处理应用的处理结果(例如，文本分类、文本序列标注、翻译等)。
示例性地，用户设备可以接收用户输入的待处理文本，然后向数据处理设备发起请求，使得数据处理设备对该待处理文本进行分类，从而得到针对该待处理文本的分类结果。其中，分类结果可以是指该待处理文本所指示的用户语义意图，比如，用户用于指示放歌、 设置时间、开启导航的意图；或者，分类结果还可以用于指示用户的情感分类结果，比如，分类结果可以指示待处理文本对应的用户情感分类为抑郁、开心或者生气等。
例如，在图1中数据处理设备可以执行本申请实施例的文本处理方法。
图2示出了另一种自然语言处理系统，在图2中，用户设备直接作为数据处理设备，该用户设备能够直接接收来自用户的输入并直接由用户设备本身的硬件进行处理，具体过程与图1相似，可参考上面的描述，在此不再赘述。
在图2所示的自然语言处理系统中，用户设备可以接收用户的指令，由用户设备自身对待处理文本进行分类得到待处理文本的分类结果。
在图2所示的自然语言处理系统中，用户设备可以接收用户的指令，例如用户设备可以接收用户输入的一段文本，然后再由用户设备自身针对该一段文本执行自然语言处理应用(例如，文本分类、文本序列标注、翻译等)，从而得到针对该一段文本的对应的自然语言处理应用的处理结果(例如，文本分类、文本序列标注、翻译等)。
在图2中，用户设备自身就可以执行本申请实施例的文本处理方法。
图3是本申请实施例提供的自然语言处理的相关设备的示意图。
上述图1和图2中的用户设备具体可以是图3中的本地设备130或者本地设备120，图1中的数据处理设备具体可以是图3中的执行设备110，其中，数据存储系统150可以存储执行设备110的待处理数据，数据存储系统150可以集成在执行设备110上，也可以设置在云上或其它网络服务器上。
图1和图2中的处理器可以通过神经网络模型或者其它模型进行数据训练/机器学习/深度学习，并利用数据最终训练或者学习得到的模型对输入的待处理文本进行处理，从而得到待处理文本处理结果。
由于本申请实施例涉及大量神经网络的应用，为了便于理解，下面先对本申请实施例可能涉及的神经网络的相关术语和概念进行介绍。
(1)神经网络
神经网络可以是由神经单元组成的，神经单元可以是指以x s和截距1为输入的运算单元，该运算单元的输出可以为：
Figure PCTCN2020135633-appb-000001
其中，s＝1、2、……n，n为大于1的自然数，W s为x s的权重，b为神经单元的偏置。f为神经单元的激活函数(activation functions)，用于将非线性特性引入神经网络中，来将神经单元中的输入信号转换为输出信号。该激活函数的输出信号可以作为下一层卷积层的输入，激活函数可以是sigmoid函数。神经网络是将多个上述单一的神经单元联结在一起形成的网络，即一个神经单元的输出可以是另一个神经单元的输入。每个神经单元的输入可以与前一层的局部接受域相连，来提取局部接受域的特征，局部接受域可以是由若干个神经单元组成的区域。
(2)深度神经网络
深度神经网络(deep neural network，DNN)，也称多层神经网络，可以理解为具有多层隐含层的神经网络。按照不同层的位置对DNN进行划分，DNN内部的神经网络可以分为三类：输入层，隐含层，输出层。一般来说第一层是输入层，最后一层是输出层，中间的层数都是隐含层。层与层之间是全连接的，也就是说，第i层的任意一个神经元一定 与第i+1层的任意一个神经元相连。
虽然DNN看起来很复杂，但是就每一层的工作来说，其实并不复杂，简单来说就是如下线性关系表达式：
Figure PCTCN2020135633-appb-000002
其中，
Figure PCTCN2020135633-appb-000003
是输入向量，
Figure PCTCN2020135633-appb-000004
是输出向量，
Figure PCTCN2020135633-appb-000005
是偏移向量，W是权重矩阵(也称系数)，α()是激活函数。每一层仅仅是对输入向量
Figure PCTCN2020135633-appb-000006
经过如此简单的操作得到输出向量
Figure PCTCN2020135633-appb-000007
由于DNN层数多，系数W和偏移向量
Figure PCTCN2020135633-appb-000008
的数量也比较多。这些参数在DNN中的定义如下所述：以系数W为例：假设在一个三层的DNN中，第二层的第4个神经元到第三层的第2个神经元的线性系数定义为
Figure PCTCN2020135633-appb-000009
上标3代表系数W所在的层数，而下标对应的是输出的第三层索引2和输入的第二层索引4。
综上，第L-1层的第k个神经元到第L层的第j个神经元的系数定义为
Figure PCTCN2020135633-appb-000010
需要注意的是，输入层是没有W参数的。在深度神经网络中，更多的隐含层让网络更能够刻画现实世界中的复杂情形。理论上而言，参数越多的模型复杂度越高，“容量”也就越大，也就意味着它能完成更复杂的学习任务。训练深度神经网络的也就是学习权重矩阵的过程，其最终目的是得到训练好的深度神经网络的所有层的权重矩阵(由很多层的向量W形成的权重矩阵)。
(3)卷积神经网络
卷积神经网络(convolutional neuron network，CNN)是一种带有卷积结构的深度神经网络。卷积神经网络包含了一个由卷积层和子采样层构成的特征抽取器，该特征抽取器可以看作是滤波器。卷积层是指卷积神经网络中对输入信号进行卷积处理的神经元层。在卷积神经网络的卷积层中，一个神经元可以只与部分邻层神经元连接。一个卷积层中，通常包含若干个特征平面，每个特征平面可以由一些矩形排列的神经单元组成。同一特征平面的神经单元共享权重，这里共享的权重就是卷积核。共享权重可以理解为提取图像信息的方式与位置无关。卷积核可以以随机大小的矩阵的形式初始化，在卷积神经网络的训练过程中卷积核可以通过学习得到合理的权重。另外，共享权重带来的直接好处是减少卷积神经网络各层之间的连接，同时又降低了过拟合的风险
(4)损失函数
在训练深度神经网络的过程中，因为希望深度神经网络的输出尽可能的接近真正想要预测的值，所以可以通过比较当前网络的预测值和真正想要的目标值，再根据两者之间的差异情况来更新每一层神经网络的权重向量(当然，在第一次更新之前通常会有初始化的过程，即为深度神经网络中的各层预先配置参数)，比如，如果网络的预测值高了，就调整权重向量让它预测低一些，不断地调整，直到深度神经网络能够预测出真正想要的目标值或与真正想要的目标值非常接近的值。因此，就需要预先定义“如何比较预测值和目标值之间的差异”，这便是损失函数(loss function)或目标函数(objective function)，它们是用于衡量预测值和目标值的差异的重要方程。其中，以损失函数举例，损失函数的输出值(loss)越高表示差异越大，那么深度神经网络的训练就变成了尽可能缩小这个loss的过程。
(5)反向传播算法
神经网络可以采用误差反向传播(back propagation，BP)算法在训练过程中修正初始的神经网络模型中参数的大小，使得神经网络模型的重建误差损失越来越小。具体地，前向传递输入信号直至输出会产生误差损失，通过反向传播误差损失信息来更新初始的神经 网络模型中参数，从而使误差损失收敛。反向传播算法是以误差损失为主导的反向传播运动，旨在得到最优的神经网络模型的参数，例如，权重矩阵。
(6)自然语言处理(natural language processing，NLP)
自然语言(natural language)即人类语言，自然语言处理(NLP)就是对人类语言的处理。自然语言处理是以一种智能与高效的方式，对文本数据进行系统化分析、理解与信息提取的过程。通过使用NLP及其组件，我们可以管理非常大块的文本数据，或者执行大量的自动化任务，并且解决各式各样的问题，如自动摘要(automatic summarization)，机器翻译(machine translation，MT)，命名实体识别(named entity recognition，NER)，关系提取(relation extraction，RE)，信息抽取(information extraction，IE)，情感分析，语音识别(speech recognition)，问答系统(question answering)以及主题分割等等。
(7)语言模型(language model，LM)
语言模型是NPL中的基础模型，通过大量语料训练学习，使得LM能够根据已有的信息(例如上下文中已经出现过的词等文本信息)来推测未知词的概率，也可以理解为LM是用来计算一个句子的概率模型。
换句话说，语言模型是自然语言文本序列的概率分布，表征特定长度特定序列文本存在的可能性。简而言之，语言模型即是根据上下文去预测下一个词是什么，由于不需要人工标注语料，因此语言模型能够从无限制的大规模语料中学习到丰富的语义知识。
语言模型主要包括统计语言模型和神经网络语言模型。
神经语言模型能够利用文本中的语义信息，对下一个单词的预测以及候选词排序提供丰富的背景知识。
统计语言模型建模直观，在中短文本语义信息的提取上更占优势。
(8)光学字符识别(optical character recognition，OCR)
OCR主要以扫描仪或相机等为输入，对其上的文字信息进行检测，通过字符识别算法对目标区域的文字信息进行识别，转换为文本格式，以便于后续的文字处理。通常，OCR系统包含有图像预处理、图像二值化、噪声去除、图像增强、图像校正、版面分析、字符切割、字符识别、版面恢复、后处理等步骤。
其中，本申请实施例中的文本处理方法应用于后处理步骤中。
首先，介绍本申请实施例提供的文本处理模型的训练方法和文本处理方法的系统架构。参考图4，本申请实施例提供了一种系统架构200。如图4中的系统架构200所示，数据采集设备260用于采集训练数据。
例如，本申请实施例中训练数据可以是训练文本处理模型的训练文本。
在采集到训练数据之后，数据采集设备260将这些训练数据存入数据库230，训练设备220基于数据库230中维护的训练数据训练得到目标模型/规则201(即本申请实施例中的一种文本处理模型)。
另外，该目标模型/规则201能够用于实现本申请实施例提供的文本处理方法，即，将待处理文本通过相关预处理(可以采用预处理模块213和/或预处理模块214进行处理)后输入该目标模型/规则201中进行处理，即可得到与文本处理模型所执行的目标任务对应的处理结果。
示例性地，目标任务是文本纠错，则文本处理模型可以为文本纠错模型，待处理文本 输入目标模型/规则201(即本申请的文本处理模型)中进行文本纠错处理，即可得到对待处理文本的纠错文本。
示例性地，目标任务是文本翻译，则文本处理模型可以文本翻译模型，待处理文本输入目标模型/规则201(即本申请的文本处理模型)中进行翻译处理，即可得到对待处理文本的翻译文本。
在本申请提供的实施例中，该目标模型/规则201是通过训练原始处理模型得到的。需要说明的是，在实际的应用中，所述数据库230中维护的训练数据不一定都来自于数据采集设备260的采集，也有可能是从其他设备接收得到的。
另外需要说明的是，训练设备220也不一定完全基于数据库230维护的训练数据进行目标模型/规则201的训练，也有可能从云端或其他地方获取训练数据进行模型训练，上述描述不应该作为对本申请实施例的限定。还需要说明的是，数据库230中维护的训练数据中的至少部分数据也可以用于执行设210对待处理文本进行处理的过程。
根据训练设备220训练得到的目标模型/规则201可以应用于不同的系统或设备中，如应用于图4所示的执行设备210，所述执行设备210可以是终端，如手机终端，平板电脑，笔记本电脑，增强现实(augmented reality，AR)/虚拟现实(virtual reality，VR)，车载终端等，还可以是服务器或者云端等。
在图4中，执行设备210配置输入/输出(input/output，I/O)接口212，用于与外部设备进行数据交互，用户可以通过客户设备240向I/O接口212输入数据，所述输入数据在本申请实施例中可以包括：待处理文本。
预处理模块213和/或预处理模块214用于根据I/O接口212接收到的输入数据进行预处理，在本申请实施例中，也可以没有预处理模块213和预处理模块214(也可以只有其中的一个预处理模块)，而直接采用计算模块211对输入数据进行处理。需要说明的是，预处理模块213或预处理模块214可以对全部的输入数据进行预处理，也可以对输入数据的部分数据进行预处理。
需要说明的是，预处理模块113和/或预处理模块214也可以是在训练设备220中训练好的。计算模块211可以用于根据上述目标模型/规则201对来自预处理模块213或者I/O接口212的输入数据执行计算等相关的处理。
在执行设备210对输入数据进行预处理，或者在执行设备210的计算模块211执行计算等相关的处理过程中，执行设备210可以调用数据存储系统250中的数据、代码等以用于相应的处理，也可以将相应处理得到的数据、指令等存入数据存储系统250中。
最后，I/O接口212将处理结果(例如纠错结果、翻译结果等)反馈给客户设备240。应理解，对应于不同的自然语言处理任务，目标模型/规则201是不同的，其处理结果相应地也是不同的。
值得说明的是，训练设备220可以针对不同的下游系统，生成该下游系统对应的目标模型/规则201，该相应的目标模型/规则201即可以用于实现上述目标或完成上述任务，从而为用户提供所需的结果。需要说明的是，训练设备220还可以针对不同的下游系统对应的目标模型/规则201生成对应的预处理模型，例如预处理模块213和/或预处理模块214中对应的预处理模型等。
在图4中所示情况下，用户可以手动给定输入数据(例如，待处理文本)，该手动给 定可以通过I/O接口212提供的界面进行操作。另一种情况下，客户设备240可以自动地向I/O接口212发送输入数据(例如，待处理文本)，如果要求客户设备240自动发送输入数据需要获得用户的授权，则用户可以在客户设备240中设置相应权限。用户可以在客户设备240查看执行设备210输出的结果，具体的呈现形式可以是显示、声音、动作等具体方式。客户设备240也可以作为数据采集端，采集如图所示输入I/O接口212的输入数据及输出I/O接口212的输出结果作为新的样本数据，并存入数据库230。当然，也可以不经过客户设备240进行采集，而是由I/O接口212直接将如图所示输入I/O接口212的输入数据及输出I/O接口212的输出结果，作为新的样本数据存入数据库230。
值得注意的是，图4仅是本申请实施例提供的一种系统架构的示意图，图中所示设备、器件、模块等之间的位置关系不构成任何限制。例如，在图4中，数据存储系统250相对执行设备210是外部存储器，在其它情况下，也可以将数据存储系统250置于执行设备210中。
如图4所示，根据训练设备220训练得到目标模型/规则201，该目标模型/规则201可以是本申请实施例中的目标处理模型，具体的，本申请实施例提供的目标处理模型可以是神经网络模型。例如可以是CNN，深度卷积神经网络(deep convolutional neural network，DCNN)。
由于CNN是一种非常常见的神经网络，下面结合图5重点对CNN的结构进行详细的介绍。如上文的基础概念介绍所述，卷积神经网络是一种带有卷积结构的深度神经网络，是一种深度学习(deep learning)架构，深度学习架构是指通过机器学习的算法，在不同的抽象层级上进行多个层次的学习。作为一种深度学习架构，CNN是一种前馈(feed-forward)人工神经网络，该前馈人工神经网络中的各个神经元可以对输入其中的图像作出响应。
如图5所示，卷积神经网络(CNN)300可以包括输入层310，卷积层/池化层320(其中池化层为可选的)，以及神经网络层330。下面对这些层的相关内容做详细介绍。
卷积层/池化层320：
卷积层：
如图5所示卷积层/池化层320可以包括如示例321-326层，举例来说：在一种实现中，321层为卷积层，322层为池化层，323层为卷积层，324层为池化层，325为卷积层，326为池化层；在另一种实现方式中，321、322为卷积层，323为池化层，324、325为卷积层，326为池化层。即卷积层的输出可以作为随后的池化层的输入，也可以作为另一个卷积层的输入以继续进行卷积操作。
下面将以卷积层321为例，介绍一层卷积层的内部工作原理。
卷积层321可以包括很多个卷积算子，卷积算子也称为核，其在自然语言处理中的作用相当于一个从输入的语音或语义信息中提取特定信息的过滤器，卷积算子本质上可以是一个权重矩阵，这个权重矩阵通常被预先定义。
这些权重矩阵中的权重值在实际应用中需要经过大量的训练得到，通过训练得到的权重值形成的各个权重矩阵可以从输入数据中提取信息，从而帮助卷积神经网络300进行正确的预测。
当卷积神经网络300有多个卷积层的时候，初始的卷积层(例如321)往往提取较多 的一般特征，该一般特征也可以称之为低级别的特征；随着卷积神经网络300深度的加深，越往后的卷积层(例如326)提取到的特征越来越复杂，比如高级别的语义之类的特征，语义越高的特征越适用于待解决的问题。
池化层：
由于常常需要减少训练参数的数量，因此卷积层之后常常需要周期性的引入池化层，即如图5中320所示例的321-326各层，可以是一层卷积层后面跟一层池化层，也可以是多层卷积层后面接一层或多层池化层。在自然语言数据处理过程中，池化层的唯一目的就是减少数据的空间大小。
神经网络层330：
在经过卷积层/池化层320的处理后，卷积神经网络300还不足以输出所需要的输出信息。因为如前所述，卷积层/池化层320只会提取特征，并减少输入数据带来的参数。然而为了生成最终的输出信息(所需要的类信息或别的相关信息)，卷积神经网络300需要利用神经网络层330来生成一个或者一组所需要的类的数量的输出。因此，在神经网络层330中可以包括多层隐含层(如图5所示的331、332至33n)以及输出层340，该多层隐含层中所包含的参数可以根据具体的任务类型的相关训练数据进行预先训练得到，例如该任务类型可以包括语音或语义识别、分类或生成等等。
在神经网络层330中的多层隐含层之后，也就是整个卷积神经网络300的最后层为输出层340，该输出层340具有类似分类交叉熵的损失函数，具体用于计算预测误差，一旦整个卷积神经网络300的前向传播(如图5由310至340的传播为前向传播)完成，反向传播(如图5由340至310的传播为反向传播)就会开始更新前面提到的各层的权重值以及偏差，以减少卷积神经网络300的损失及卷积神经网络300通过输出层输出的结果和理想结果之间的误差。
需要说明的是，如图5所示的卷积神经网络300仅作为一种卷积神经网络的示例，在具体的应用中，卷积神经网络还可以以其他网络模型的形式存在。
如图6所示，卷积神经网络(CNN)300可以包括输入层310，卷积层/池化层320(其中池化层为可选的)，以及神经网络层330，在图6中，卷积层/池化层320中的多个卷积层/池化层并行，将分别提取的特征均输入给全神经网络层330进行处理。
图7为本申请实施例提供的一种芯片的硬件结构的示意图。该芯片包括神经网络处理器(neural processing unit，NPU)40。该芯片可以被设置在如图4所示的执行设备110中，用以完成计算模块111的计算工作。该芯片也可以被设置在如图4所示的训练设备120中，用以完成训练设备120的训练工作并输出目标模型/规则101。如图5和图6所示的卷积神经网络中各层的算法均可在如图7所示的芯片中得以实现。
NPU 40作为协处理器可以挂载到主CPU(host CPU)上，由主CPU分配任务。NPU 40的核心部分为运算电路403，在NPU 40工作时，NPU 40中的控制器404可以控制运算电路403提取存储器(权重存储器或输入存储器)中的数据并进行运算。
在一些实现中，运算电路403内部包括多个处理单元(process engine，PE)。在一些实现中，运算电路403是二维脉动阵列。运算电路403还可以是一维脉动阵列或者能够执行例如乘法和加法这样的数学运算的其它电子线路。在一些实现中，运算电路403是通用的矩阵处理器。
举例来说，假设有输入矩阵A，权重矩阵B，输出矩阵C。运算电路从权重存储器402中取矩阵B相应的数据，并缓存在运算电路中每一个PE上。运算电路从输入存储器401中取矩阵A数据与矩阵B进行矩阵运算，得到的矩阵的部分结果或最终结果，保存在累加器(accumulator)408中。
向量计算单元407可以对运算电路的输出做进一步处理，如向量乘，向量加，指数运算，对数运算，大小比较等等。例如，向量计算单元407可以用于神经网络中非卷积/非全连接层(fully connected layers，FC)层的网络计算，如池化(pooling)，批归一化(batch normalization)，局部响应归一化(local response normalization)等。
在一些实现中，向量计算单元407能将经处理的输出的向量存储到统一缓存器406。例如，向量计算单元407可以将非线性函数应用到运算电路403的输出，例如累加值的向量，用以生成激活值。在一些实现中，向量计算单元407生成归一化的值、合并值，或二者均有。在一些实现中，处理过的输出的向量能够用作到运算电路403的激活输入，例如用于在神经网络中的后续层中的使用。
统一存储器406用于存放输入数据以及输出数据。
权重数据直接通过存储单元访问控制器405(direct memory access controller，DMAC)将外部存储器中的输入数据搬运到输入存储器401和/或统一存储器406、将外部存储器中的权重数据存入权重存储器402，以及将统一存储器406中的数据存入外部存储器。
总线接口单元(bus interface unit，BIU)410，用于通过总线实现主CPU、DMAC和取指存储器409之间进行交互。
与控制器404连接的取指存储器(instruction fetch buffer)409，用于存储控制器404使用的指令；
控制器404，用于调用指存储器409中缓存的指令，实现控制该运算加速器的工作过程。
一般地，统一存储器406，输入存储器401，权重存储器402以及取指存储器409均可以为片上(on-chip)存储器。NPU的外部存储器可以为该NPU外部的存储器，该外部存储器可以为双倍数据率同步动态随机存储器(double data rate synchronous dynamic random access memory，DDR SDRAM)、高带宽存储器(high bandwidth memory，HBM)或其他可读可写的存储器。应理解，图7示出的芯片硬件结果仅为示例性说明，本申请并未限定于此。
图8为本申请实施例提供的一种翻译场景下的系统结构的示意图。如图8所示，本申请实施例中的文本处理方法可以由自然语言理解(natural language understand，NLU)云侧模块执行。
该系统包括视觉模块、OCR引擎模块、OCR识别模块、NLU模块、NLU云侧模块、翻译模块和翻译云模块。
视觉模块：用于采集图片。例如，视觉模块可以通过拍照等方式采集图片。
OCR引擎模块：用于OCR任务的调度。
OCR识别模块：用于基于OCR算法实现字符的识别。
NLU模块：用于NLU相关任务的调度。
NLU云侧模块：用于对接收到的文本中的错词/语法进行纠错。
翻译模块：用于多种语言间的翻译任务的调度。
翻译云模块：用于对接收到的文本进行翻译。
下面结合图8对本申请实施例中的文本处理方法应用于翻译场景进行详细介绍。
S1：视觉模块将采集的图片传输至OCR引擎模块。
S2：OCR引擎模块通过调度将图片传输至OCR识别模块。
S3：OCR识别模块识别出图片中的文字，即原文，将原文返回至OCR引擎模块。
S4：OCR引擎模块将原文传输至NLU模块。
S5：NLU模块通过调度将原文传输至NLU云侧模块。
S6：NLU云侧模块对原文中的错词/语法进行纠错，得到纠错后的原文。
S7：NLU云侧模块将纠错后的原文返回至NLU模块。
S8：NLU模块将纠错后的原文回传至OCR引擎模块。
S9：OCR引擎模块将纠错后的原文传输至翻译模块。
S10：翻译模块通过调度将纠错后的原文传输至翻译云模块。
S11：翻译云模块进行翻译，得到译文，并回传至翻译模块。
S12：翻译模块将译文回传至OCR引擎模块。
该系统中，文本处理方法用于文本纠错，也就是文本处理模型可以为文本纠错模型。将待处理文本输入文本处理模型中进行纠错处理，即可得到对待处理文本的纠正结果。应理解，图8仅为本申请实施例中的文本处理方法的一种示例，在另一种可能的实现方式中，文本处理模型可以文本翻译模型，将待处理文本输入文本翻译模型中进行纠错处理，并将纠错后的结果进行翻译处理，即可得到对待处理文本的翻译文本。
应理解，以上仅为本申请实施例中的文本处理模型的示例。
图8中的文本处理模型部署在云服务器上，应理解，文本处理模型也可以部署在智能终端设备上。智能终端可以是具有摄像头的电子设备，例如，智能终端可以是有图像处理功能的移动电话、平板个人电脑(tablet personal computer，TPC)、媒体播放器、智能电视、笔记本电脑(laptop computer，LC)、个人数字助理(personal digital assistant，PDA)、个人计算机(personal computer，PC)或者，自动驾驶车辆中的车载终端等，本申请实施例对此不作限定。
图9是本申请实施例提供的文本处理方法的示意性流程图。图9所示文本处理方法可以由文本处理装置执行，该装置具体可以是图1中的数据处理设备，也可以是图2中的用户设备，也可以是图3中的执行设备110或者本地设备，也可以是图4中的执行设备210。图8所示的方法包括步骤510至550，下面分别对步骤510至550进行详细的介绍。
510，获取待处理文本。
待处理文本可以是OCR输出文本，或者，可以是用户输入的文本。例如，用户输入的文本可以包括社交网络中发布的内容，或者可以是搜索引擎的搜索框中输入的内容等。应理解，待处理文本可以是任意需要进行纠错的文本，本申请实施例对待处理文本的具体形式不做限定。
520，对待处理文本进行检错处理，得到待处理文本中的错词。
待处理文本中的错误可以包括非词错误(non-word error)和真词错误(real-word error)。非词错误是指待处理文本中的词不在预设词库中。真词错误是指待处理文本中的词存在于 预设词库中，但导致上下文语义出现问题，不是当前语境所需要的词。相应地，错词可以包括非词和错误真词。
示例性地，可以基于第一预设词库对待处理文本进行非词错误检测，得到待处理文本中的非词。第一预设词库能够用于区分真词和非词。真词指的是存在于第一预设词库中的词，相对应地，非词指的是不存在于第一预设词库中的词。
能够用于检测非词错误的词库均可以理解为第一预设词库。例如，在方法500应用于英文文本纠错的情况下，第一预设词库可以为英文字典。非词即为不存在于英文字典的词，例如，werld。本申请实施例对第一预设词库的类型不做限定。
应理解，在本申请实施例中，“词库”也可以称为“字典”或“词表”等。
示例性地，可以基于语言模型对待处理文本进行真词错误检测，得到待处理文本中的错误真词。例如，当一个词对应的文本困惑度(perplexity)高于设定阈值，判断该词为错误真词。
例如，该语言模型可以为统计语言模型，例如，n-gram模型。
统计语言模型对中短文本语义信息的提取更占优势，适用于较少依赖长距离的语义信息的场景，例如，OCR场景下的文本纠错。
再例如，该语言模型也可以为神经网络模型，例如，循环神经网络(recurrent neural network，RNN)模型。
应理解，步骤520中的错词可以仅包括非词，也可以仅包括错误真词，还可以包括非词和错误真词。
530，根据预先存储的错词候选词词库生成错词对应的候选词集。错词候选词词库用于指示多个错词对应的候选词集。
一个错词可以对应一个或多个候选词。一个候选词也可以对应一个或多个错词。也就是说，一个错词对应的候选词集中可以仅包括一个候选词。
根据预先存储的错词候选词词库生成错词对应的候选词集中的候选词可以为一个，也可以为多个。
该错词候选词词库可以是离线生成的错词候选词词库。
可选地，错词包括错误真词，错词候选词词库可以包括错误真词候选词词库。在该情况下，步骤530可以包括步骤531。
531，根据预先存储的错误真词候选词词库确定错误真词对应的候选词集。错误真词候选词词库包括多个错误真词对应的候选词集。一个错误真词对应的候选词可以为一个，也可以为多个。
错误真词候选词词库可以是离线生成的。
可选地，多个错误真词对应的候选词集可以是基于多个错误真词与多个错误真词对应的候选词之间的相似性确定的。
示例性地，多个错误真词与多个错误真词对应的候选词之间的相似性可以包括多个错误真词与多个错误真词对应的候选词之间的编辑距离和/或公共字符串。也就是可以基于最小编辑距离和/或最大公共字符串确定错误真词对应的候选词。
最小编辑距离是指将一个词转换为另一个词所需的最少编辑操作次数。编辑操作包括对词中的字符的插入、删除、易位和替换等操作。
最大公共字符串是指两个词中所包含的连续相同的字符的数量。
应理解，该相似性也可以是其他形式的相似性，例如字符相似性等。
例如，错误真词候选词词库中的一个错误真词为word，该错误真词对应的候选词可以包括world、words和sword。当待处理文本中的错误真词包括word时，可以根据该错误真词候选词词库将word对应的候选词确定为world、words和sword中的至少一个，作为word的候选词集。
可选地，错词可以包括非词，错词候选词词库可以包括非词候选词词库。在该情况下，步骤530可以包括：
532，根据预先存储的非词候选词词库确定非词对应的候选词集。
非词候选词词库可以包括多个非词对应的候选词集。
例如，非词候选词词库中的一个非词为werld，werld对应的候选词可以包括world、word和sword。当待处理文本中的非词包括werld时，可以根据非词候选词词库将werld对应的候选词确定为world、word和sword中的至少一个，作为werld的候选词集。
示例性地，非词候选词词库中的多个非词对应的候选词集可以是基于多个非词与多个非词对应的候选词之间的相似性确定的。一个非词对应的候选词可以为一个，也可以为多个。也就是说，一个非词对应的候选词集中的候选词可以仅为一个。
例如，多个非词与多个非词对应的候选词之间的相似性可以包括多个非词与多个非词对应的候选词之间的编辑距离和/或公共字符串。也就是可以基于最小编辑距离和/或最大公共字符串确定非词对应的候选词。
应理解，该相似性也可以是其他形式的相似性，例如字符相似性等。
对于非词，直接穷举非词数量非常庞大，例如，word对应456976(26的4次方)个非词。如果对70万字的英文字典建立非词词库，非词的数量将达到百亿级，存储和检错消耗极大。
下面给出根据预先存储的非词候选词词库生成非词对应的候选词的另一种实现方式。
非词候选词词库中包括多个公共子词对应的候选词集。一个公共子词对应的候选词可以为一个，也可以为多个。也就是说一个公共子词的候选词集中可以仅包括一个候选词。
在该情况下，步骤532可以包括步骤532a和步骤532b。
532a，生成待处理文本中的非词对应的公共子词。
其中，该公共子词与该非词之间的相似性满足预设条件。一个非词对应的公共子词可以为一个，也可以为多个。
该公共子词与该非词之间的相似性可以包括该公共子词与该非词之间的编辑距离。预设条件可以为该公共子词与该非词之间的最大编辑距离小于预设值。例如，该预设值可以为2。
具体地，限制非词和该非词对应的公共子词的最大编辑距离为2，生成待处理文本中的非词对应的公共子词。也就是由非词生成该非词对应的公共子词的过程中操作次数不超过两次。
例如，对于非词wored，基于非词和该非词对应的公共子词的最大编辑距离为2，得到ored、wred、woed等15个公共子词。
可替换地，在待处理文本为OCR输出文本的情况下，生成该非词对应的公共子词的 方式可以由OCR对一个字母识别错误的概率确定。例如，OCR将o识别为e的概率较大，对于非词werd，可以将该字母e删除，得到werd对应的公共子词wrd。
532b，根据非词候选词词库将待处理文本中的非词对应的公共子词对应的候选词集确定为待处理文本中的非词对应的候选词集。
也就是说公共子词相当于候选词和非词之间的桥梁，通过公共子词能够间接生成非词对应的候选词。
非词候选词词库可以是离线生成的。
具体地，非词候选词词库中的多个公共子词对应的候选词集可以是基于多个公共子词与多个公共子词对应的候选词之间的相似性确定的。
示例性地，多个公共子词与多个非词对应的候选词之间的相似性可以包括多个公共子词与多个公共子词对应的候选词之间的编辑距离和/或公共字符串。也就是可以基于最小编辑距离和/或最大公共字符串确定公共子词对应的候选词。
应理解，该相似性也可以是其他形式的相似性，例如字符相似性等。
70万单词的公共子词的规模为千万级，而在没有公共子词的情况下，对70万单词建立非词候选词词库，非词的数量将达到百亿级。通过设置公共子词，明显减少了非词候选词词库所需的存储空间。对于长度为n的词，在线生成该词的候选词的过程中，能够得到k个词，k为26的n次方，在k个词中得到该词的候选词。而先在线生成该词的公共子词，公共子词的数量仅为n+n*(n-1)/2，然后根据公共子词对应的候选词确定该词的候选词。显然，通过在线生成该词的公共子词，进而确定非词的候选词，能够显著降低文本纠错的时间消耗。
应理解，步骤530中的非词可以是预处理之后的非词。该预处理可以包括过滤掉特殊格式的词。特殊格式的词可以是满足预设标准的词。例如，特殊格式的词可以包括合并词。合并词指的是至少两个单词由于缺少空格形成的一个单词。比如，inChina。
本申请实施例中对预处理的方式不做限定。
540，在错词对应的候选词集中确定错词对应的目标候选词。
具体地，在错词对应的候选词集中确定错词对应的目标候选词，可以为，在错词对应的候选词中随机确定错词对应的目标候选词。
可选地，步骤540可以包括步骤541和步骤542。
541，根据错词对应的候选词集中的候选词与错词之间的相似性以及错词对应的候选词集中的候选词的困惑度对错词对应的候选词集中的候选词进行评分。
其中，错词对应的候选词集中的候选词的困惑度用于指示错词对应的候选词集中的候选词在待处理文本中出现的可能性。
错词对应的候选词集中的候选词与错词之间的相似性可以包括：错词对应的候选词集中的候选词与错词之间的形态相似性以及错词对应的候选词集中的候选词与错词之间的编辑距离。
形态相似性用于衡量两个词在形态特征上的相似性。例如，可以基于最大公共字符串和字符相似性对错词和错词对应的候选词集中的候选词进行形态特征上相似性的判断，也就是得到形态性相似性对应的评分。
错词对应的候选词集中的候选词的困惑度可以通过语言模型进行评分。
每个候选词对应的评分可以由上述几项对应的评分进行加权得到，也就是为每一项对应的评分设置权重。该权重可以是预先设定的，也可以是训练得到的。
根据相似性和文本的困惑度进行评分，同时考虑了错词与候选词的相似性以及待处理文本的语义信息，能够得到较优的候选词，能够得到更准确的评分结果。
542，将错词对应的候选词集中评分最高的候选词确定为该错词对应的目标候选词。
应理解，步骤540中，错词可以包括非词和错误真词。对于非词和错误真词可以采用不同的方式确定目标候选词，也可以采用相同的方式确定目标候选词。
例如，对于待处理文本中的非词，可以执行步骤541和步骤542，得到待处理文本中的非词对应的候选词。
对于待处理文本中的错误真词，在待处理文本中的错误真词对应的候选词中随机确定待处理文本中的错误真词对应的目标候选词。
550，根据错词对应的目标候选词对错词进行校正。
对错词进行校正可以包括利用错词对应的目标候选词替换错词，也包括不对错词进行处理，也就是不对错词进行替换。
示例性地，步骤550可以为步骤550a。
550a，直接利用错词对应的目标候选词替换错词，作为文本纠正的结果。
可替换地，步骤550可以为步骤550b。
550b，判断错词对应的目标候选词与错词之间的形态相似性，在形态相似性高于或等于预设阈值的情况下，利用错词对应的目标候选词替换错词，作为错词的校正结果。
进一步地，在形态相似性低于预设阈值的情况下，不对错词进行纠正。也就是不利用错词对应的目标候选词替代错词。这样可以降低时间消耗，快速实现文本纠错。
可替换地，在形态相似性低于预设阈值的情况下，可以将步骤541中评分第二高的候选词作为错词对应的目标候选词，重复步骤550b，直至得到形态相似性满足预设阈值的目标候选词，并利用目标候选词替换错词。
这样，可以避免将原本识别正确的词替换为错误的词。例如，对于OCR场景，OCR输出的文本应该与正确的文本具备较高的形态相似性，当错词与错词对应的目标候选词的形态相似性低于一定阈值时，也就是两者的形态相似性较低，在该情况下，不对错词进行纠正，避免引入新的错误。
可替换地，步骤550可以为步骤550c。
550c，通过语言模型检测包含错词对应的目标候选词的文本的困惑度，在困惑度低于困惑度阈值的情况下，利用错词对应的目标候选词替换错词，作为错词的校正结果。
进一步地，在困惑度高于预定阈值的情况下，不对错词进行纠正。也就是不利用错词对应的目标候选词替代错词。这样可以降低时间消耗，快速实现文本纠错。
可替换地，在困惑度高于预定阈值的情况下，可以将步骤541中评分第二高的候选词作为错词对应的目标候选词，重复步骤550c，直至得到困惑度满足预定阈值的目标候选词，并利用目标候选词替换错词。
应理解，在步骤550中，对于待处理文本中的非词和待处理文本中的错误真词可以分别执行不同的步骤，也可以执行相同的步骤。例如，对于待处理文本中的非词，可以执行步骤550a，对于待处理文本中的错误真词，可以执行步骤550b，也可以执行550a。也就 是说，对于待处理文本中的非词和待处理文本中的错误真词可以分别执行步骤550，也可以共同执行步骤550。
在本申请实施例中，根据预设的错词候选词词库，能够快速生成错词对应的候选词，计算量较小，降低了文本纠错的时间消耗，保证了文本处理的实时性。此外，采用多种方式对候选词进行评分，得到较优的候选词，提高了文本纠错的准确性。同时，通过错词对应的目标候选词与错词的形态相似性判断，利用形态相似性高于或等于预设阈值的目标候选词对错词进行纠正，能够避免引入新的错误，例如，将原本识别正确的修改为错误的词。
图10是本申请实施例提供的文本处理方法600的流程示意图。该方法600可以是方法500的一个具体例子。方法600包括步骤610至步骤6140。下面对步骤610至步骤6140进行详细说明。
610，获取待处理文本。
进一步地，方法600还可以包括步骤611。
611，判断待处理文本的长度，在待处理文本的长度大于或等于预设长度的情况下，执行步骤620。其中，待处理文本的长度指的是待处理文本中的词的数量。例如，预设长度可以为2。
这样保证了待处理文本中的词的数量，能够更好地利用上下文语义信息，提高了文本纠错的准确率。
620，基于英文词库对该待处理文本进行非词错误检测。该英文词库为方法500中的第一预设词库的一例。
步骤620用于得到待处理文本中的非词和真词。非词为不存在于该英文词库中的词。真词为存在于该英文词库中的词。
对于待处理文本中的非词执行步骤630。对于待处理文本中的真词执行690。
为了便于描述，将待处理文本中的非词称为非词1#。非词1#可以包括一个非词，也可以包括多个非词。
630，生成非词1#的对应的公共子词1#。公共子词与1#与非词1#之间的相似性满足预设条件。
公共子词1#指的是非词1#对应的公共子词。公共子词1#可以为一个，也可以为多个。
公共子词1#与非词1#之间的相似性可以包括公共子词1#与非词1#之间的编辑距离。预设条件可以为公共子词1#与非词1#之间的最大编辑距离小于预设值。例如，该预设值可以为2。
具体地，限制待处理文本中的非词1#和公共子词1#的最大编辑距离为2，生成公共子词1#。
640，获取公共子词候选词词库。公共子词候选词词库为方法500中的非词候选词词库的一例。该公共子词候选词词库包括多个公共子词对应的候选词集。
650，确定非词1#对应的候选词集。
具体地，根据公共子词候选词词库将公共子词1#对应的候选词集确定为非词1#对应的候选词集。
660，对非词1#对应的候选词集中的候选词进行评分。
示例性地，可以通过语言模型、编辑距离和最大公共字符串等对非词1#对应的候选 词集中的候选词进行评分。详细过程如方法500中的步骤541所述，此处不再赘述。
670，确定非词1#对应的目标候选词。
示例性地，可以将步骤660中评分最高的候选词作为非词1#对应的目标候选词。
680，判断非词1#对应的目标候选词与非词1#之间的形态相似性。若非词1#对应的目标候选词与非词1#之间的形态相似性高于或等于预设阈值，则利用非词1#对应的目标候选词替换非词1#，即将非词1#对应的目标候选词作为非词1#的校正结果。
可选地，若非词1#对应的目标候选词与非词1#之间的形态相似性低于预设阈值，则将非词1#作为非词1#的校正结果，即不对非词1#进行处理。
690，基于语言模型对待处理文本中的真词进行真词错误检测，得到待处理文本中的错误真词。
例如，当一个词对应的文本困惑度高于设定阈值，判断该词为错误真词。
对于待处理文本中的错误真词执行步骤6100。为了便于描述，将待处理文本中的错误真词称为错误真词1#。错误真词1#可以包括一个错误真词，也可以包括多个错误真词。
6100，获取错误真词候选词词库。该错误真词候选词词库包括多个错误真词对应的候选词集。
6110，确定错误真词1#对应的候选词集。
具体地，根据错误真词候选词词库确定错误真词1#对应的候选词集。
6120，对错误真词1#对应的候选词集中的候选词进行评分。
示例性地，可以通过语言模型、编辑距离和最大公共字符串等对错误真词1#对应的候选词集中的候选词进行评分。详细过程如方法500中的步骤541所述，此处不再赘述。
6130，确定错误真词1#对应的目标候选词。
示例性地，可以将步骤6120中评分最高的候选词作为错误真词1#对应的目标候选词。
6140，判断错误真词1#对应的目标候选词与错误真词1#之间的形态相似性。若错误真词1#对应的目标候选词与错误真词1#之间的形态相似性高于或等于预设阈值，则利用错误真词1#对应的目标候选词替换错误真词1#，即将错误真词1#对应的目标候选词作为错误真词1#的校正结果。
可选地，若错误真词1#对应的目标候选词与错误真词1#之间的形态相似性低于预设阈值，则将错误真词1#作为错误真词1#的校正结果，即不对错误真词1#进行处理。
在本申请实施例中，根据预设的错误真词候选词词库和公共子词候选词词库，能够快速生成错误真词对应的候选词集和非词对应的候选词集，计算量较小，降低了文本纠错的时间消耗，保证了文本处理的实时性。此外，采用多种方式对候选词进行评分，能够得到较优的候选词，提高了文本纠错的准确性。同时，通过错词对应的目标候选词与错词的形态相似性判断，利用形态相似性高于或等于预设阈值的目标候选词对错词进行纠正，能够避免引入新的错误，例如，将原本识别正确的修改为错误的词。
上文结合图1至图10，详细描述了本申请实施例文本处理方法，下面将结合图11和图12，详细描述本申请的装置实施例。应理解，方法实施例的描述与装置实施例的描述相互对应，因此，未详细描述的部分可以参见前面方法实施例。
图11是本申请实施例提供的文本处理装置的示意性框图。应理解，文本处理装置1000可以执行图9或图10所示的文本处理方法。该文本处理装置1000包括：获取单元1010 和处理单元1020。
其中，获取单元1010，用于获取待处理文本；处理单元1020，用于：对待处理文本对待处理文本进行检错处理，得到待处理文本中的错词；根据预先存储的错词候选词词库确定错词对应的候选词集，错词候选词词库用于指示多个错词对应的候选词集；在错词对应的候选词集中确定错词对应的目标候选词；根据错词对应的目标候选词对错词进行校正。
可选地，处理单元1020用于：根据错词对应的候选词集中的候选词与错词之间的相似性以及错词对应的候选词集中的候选词的困惑度对错词对应的候选词集中的候选词进行评分，其中，错词对应的候选词集中的候选词的困惑度用于指示错词对应的候选词集中的候选词在待处理文本中出现的可能性；将错词对应的候选词集中评分最高的候选词确定为错词对应的目标候选词。
可选地，错词对应的候选词集中的候选词与错词之间的相似性包括错词对应的候选词集中的候选词与错词之间的形态相似性和错词对应的候选词集中的候选词与错词之间的编辑距离
可选地，错词包括非词，错词候选词词库包括非词候选词词库。
可选地，非词候选词词库包括多个公共子词对应的候选词集，以及处理单元1020用于：生成非词对应的公共子词，其中，公共子词与非词之间的相似性满足预设条件；根据非词候选词词库将公共子词对应的候选词集确定为非词对应的候选词集。
可选地，非词候选词词库中的多个公共子词对应的候选词集是基于多个公共子词与多个公共子词对应的候选词之间的相似性确定的。
可选地，错词包括错误真词，错词候选词词库包括错误真词候选词词库，错误真词候选词词库包括多个错误真词对应的候选词集。
可选地，错误真词候选词词库中的多个错误真词对应的候选词集是基于多个错误真词与多个错误真词对应的候选词之间的相似性确定的。
可选地，处理单元1020用于：判断错词对应的目标候选词和错词之间的形态相似性；在形态相似性高于或等于预设阈值的情况下，将错词对应的目标候选词作为错词的校正结果。
需要说明的是，上述文本处理装置1000以功能单元的形式体现。这里的术语“单元”可以通过软件和/或硬件形式实现，对此不作具体限定。
例如，“单元”可以是实现上述功能的软件程序、硬件电路或二者结合。所述硬件电路可能包括应用特有集成电路(application specific integrated circuit，ASIC)、电子电路、用于执行一个或多个软件或固件程序的处理器(例如共享处理器、专有处理器或组处理器等)和存储器、合并逻辑电路和/或其它支持所描述的功能的合适组件。
因此，在本申请的实施例中描述的各示例的单元，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
图12是本申请实施例提供的文本处理装置的硬件结构示意图。图12所示的文本处理装置1200(该文本处理装置1200具体可以是一种计算机设备)包括存储器1201、处理器 1202、通信接口1203以及总线1204。其中，存储器1201、处理器1202、通信接口1203通过总线1204实现彼此之间的通信连接。
存储器1201可以是只读存储器(read only memory，ROM)，静态存储设备，动态存储设备或者随机存取存储器(random access memory，RAM)。存储器1201可以存储程序，当存储器1201中存储的程序被处理器1202执行时，处理器1202用于执行本申请实施例的文本处理方法的各个步骤，例如，执行图9或图10所示的各个步骤。
应理解，本申请实施例所示的文本处理装置可以是智能终端，也可以是配置于智能终端中的芯片。
上述本申请实施例揭示的文本处理方法可以应用于处理器1202中，或者由处理器1202实现。处理器1202可能是一种集成电路芯片，具有信号的处理能力。在实现过程中，上述文本处理方法的各步骤可以通过处理器1202中的硬件的集成逻辑电路或者软件形式的指令完成。例如，处理器1202可以是包含图7所示的NPU的芯片。
上述的处理器1202可以是中央处理器(central processing unit，CPU)、图形处理器(graphics processing unit，GPU)、通用处理器、数字信号处理器(digital signal processor，DSP)、专用集成电路(application specific integrated circuit，ASIC)、现成可编程门阵列(field programmable gate array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。可以实现或者执行本申请实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。结合本申请实施例所公开的方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存取存储器(random access memory，RAM)、闪存、只读存储器(read-only memory，ROM)、可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器1201，处理器1202读取存储器1201中的信息，结合其硬件完成本申请实施中图11所示的文本处理装置中包括的单元所需执行的功能，或者，执行本申请方法实施例的图9或图10所示的文本处理方法。
通信接口1203使用例如但不限于收发器一类的收发装置，来实现装置1200与其他设备或通信网络之间的通信。
总线1204可包括在文本处理装置1200各个部件(例如，存储器1201、处理器1202、通信接口1203)之间传送信息的通路。
应注意，尽管上述文本处理装置1200仅仅示出了存储器、处理器、通信接口，但是在具体实现过程中，本领域的技术人员应当理解，文本处理装置1200还可以包括实现正常运行所必须的其他器件。同时，根据具体需要本领域的技术人员应当理解，上述文本处理装置1200还可包括实现其他附加功能的硬件器件。此外，本领域的技术人员应当理解，上述文本处理装置1200也可仅仅包括实现本申请实施例所必须的器件，而不必包括图12中所示的全部器件。
本申请实施例还提供一种芯片，该芯片包括收发单元和处理单元。其中，收发单元可以是输入输出电路、通信接口；处理单元为该芯片上集成的处理器或者微处理器或者集成电路。该芯片可以执行上述方法实施例中的方法。
本申请实施例还提供一种计算机可读存储介质，其上存储有指令，该指令被执行时执 行上述方法实施例中的方法。
本申请实施例还提供一种包含指令的计算机程序产品，该指令被执行时执行上述方法实施例中的方法。
还应理解，本申请实施例中，该存储器可以包括只读存储器和随机存取存储器，并向处理器提供指令和数据。处理器的一部分还可以包括非易失性随机存取存储器。例如，处理器还可以存储设备类型的信息。
应理解，本文中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系。
应理解，在本申请的各种实施例中，上述各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本申请实施例的实施过程构成任何限定。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统、装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能模块可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。
所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(read-only memory，ROM)、随机存取存储器(random access memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟 悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。

Claims (20)

  1. 一种文本处理方法，其特征在于，包括：
    获取待处理文本；
    对所述待处理文本进行检错处理，得到所述待处理文本中的错词；
    根据预先存储的错词候选词词库确定所述错词对应的候选词集，所述错词候选词词库用于指示多个错词对应的候选词集；
    在所述错词对应的候选词集中确定所述错词对应的目标候选词；
    根据所述错词对应的目标候选词对所述错词进行校正。
  2. 如权利要求1所述的方法，其特征在于，在所述错词对应的候选词集中确定所述错词对应的目标候选词，包括：
    根据所述错词对应的候选词集中的候选词与所述错词之间的相似性以及所述错词对应的候选词集中的候选词的困惑度对所述错词对应的候选词集中的候选词进行评分，其中，所述错词对应的候选词集中的候选词的困惑度用于指示所述错词对应的候选词集中的候选词在所述待处理文本中出现的可能性；
    将所述错词对应的候选词集中评分最高的候选词确定为所述错词对应的目标候选词。
  3. 如权利要求2所述的方法，其特征在于，所述错词对应的候选词集中的候选词与所述错词之间的相似性包括所述错词对应的候选词集中的候选词与所述错词之间的形态相似性和所述错词对应的候选词集中的候选词与所述错词之间的编辑距离。
  4. 如权利要求1至3中任一项所述的方法，其特征在于，所述错词包括非词，所述错词候选词词库包括非词候选词词库。
  5. 如权利要求4所述的方法，其特征在于，所述非词候选词词库包括多个公共子词对应的候选词集，以及
    所述根据预先存储的错词候选词词库确定所述错词对应的候选词集，包括：
    生成所述非词对应的公共子词，其中，所述公共子词与所述非词之间的相似性满足预设条件；
    根据所述非词候选词词库将所述公共子词对应的候选词集确定为所述非词对应的候选词集。
  6. 如权利要求5所述的方法，其特征在于，所述非词候选词词库中的多个公共子词对应的候选词集是基于所述多个公共子词与所述多个公共子词对应的候选词之间的相似性确定的。
  7. 如权利要求1至6中任一项所述的方法，其特征在于，所述错词包括错误真词，所述错词候选词词库包括错误真词候选词词库，所述错误真词候选词词库包括多个错误真词对应的候选词集。
  8. 如权利要求7所述的方法，其特征在于，所述错误真词候选词词库中的多个错误真词对应的候选词集是基于所述多个错误真词与所述多个错误真词对应的候选词之间的相似性确定的。
  9. 如权利要求1至8中任一项所述的方法，其特征在于，所述根据所述错词对应的 目标候选词对所述错词进行校正，包括：
    判断所述错词对应的目标候选词和所述错词之间的形态相似性；
    在所述形态相似性高于或等于预设阈值的情况下，将所述错词对应的目标候选词作为所述错词的校正结果。
  10. 一种文本处理装置，其特征在于，包括：
    获取单元，所述获取单元用于获取待处理文本；
    处理单元，所述处理单元用于：
    对所述待处理文本进行检错处理，得到所述待处理文本中的错词；
    根据预先存储的错词候选词词库确定所述错词对应的候选词集，所述错词候选词词库用于指示多个错词对应的候选词集；
    在所述错词对应的候选词集中确定所述错词对应的目标候选词；
    根据所述错词对应的目标候选词对所述错词进行校正。
  11. 如权利要求10所述的装置，其特征在于，所述处理单元用于：
    根据所述错词对应的候选词集中的候选词与所述错词之间的相似性以及所述错词对应的候选词集中的候选词的困惑度对所述错词对应的候选词集中的候选词进行评分，其中，所述错词对应的候选词集中的候选词的困惑度用于指示所述错词对应的候选词集中的候选词在所述待处理文本中出现的可能性；
    将所述错词对应的候选词集中评分最高的候选词确定为所述错词对应的目标候选词。
  12. 如权利要求11所述的装置，其特征在于，所述错词对应的候选词集中的候选词与所述错词之间的相似性包括所述错词对应的候选词集中的候选词与所述错词之间的形态相似性和所述错词对应的候选词集中的候选词与所述错词之间的编辑距离。
  13. 如权利要求10至12中任一项所述的装置，其特征在于，所述错词包括非词，所述错词候选词词库包括非词候选词词库。
  14. 如权利要求13所述的装置，其特征在于，所述非词候选词词库包括多个公共子词对应的候选词集，以及所述处理单元用于：
    生成所述非词对应的公共子词，其中，所述公共子词与所述非词之间的相似性满足预设条件；
    根据所述非词候选词词库将所述公共子词对应的候选词集确定为所述非词对应的候选词集。
  15. 如权利要求14所述的装置，其特征在于，所述非词候选词词库中的多个公共子词对应的候选词集是基于所述多个公共子词与所述多个公共子词对应的候选词之间的相似性确定的。
  16. 如权利要求10至15中任一项所述的装置，其特征在于，所述错词包括错误真词，所述错词候选词词库包括错误真词候选词词库，所述错误真词候选词词库包括多个错误真词对应的候选词集。
  17. 如权利要求16所述的装置，其特征在于，所述错误真词候选词词库中的多个错误真词对应的候选词集是基于所述多个错误真词与所述多个错误真词对应的候选词之间的相似性确定的。
  18. 如权利要求10至17中任一项所述的装置，其特征在于，所述处理单元用于：
    判断所述错词对应的目标候选词和所述错词之间的形态相似性；
    在所述形态相似性高于或等于预设阈值的情况下，将所述错词对应的目标候选词作为所述错词的校正结果。
  19. 一种文本处理装置，其特征在于，包括：
    存储器，用于存储程序；
    处理器，用于执行所述存储器存储的程序，当所述处理器执行所述存储器存储的程序时，所述处理器用于执行权利要求1至9中任一项所述的文本处理方法。
  20. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质中存储有程序指令，当所述程序指令由处理器运行时，实现权利要求1至9中任一项所述的文本处理方法。
PCT/CN2020/135633 2019-12-23 2020-12-11 文本处理方法及装置 WO2021129410A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201911335094.7A CN113095066A (zh) 2019-12-23 2019-12-23 文本处理方法及装置
CN201911335094.7 2019-12-23

Publications (1)

Publication Number Publication Date
WO2021129410A1 true WO2021129410A1 (zh) 2021-07-01

Family

ID=76572941

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/135633 WO2021129410A1 (zh) 2019-12-23 2020-12-11 文本处理方法及装置

Country Status (2)

Country Link
CN (1) CN113095066A (zh)
WO (1) WO2021129410A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117523590A (zh) * 2023-11-10 2024-02-06 广州方舟信息科技有限公司 一种校验生产厂商名称的方法、装置、设备及存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103970765A (zh) * 2013-01-29 2014-08-06 腾讯科技（深圳）有限公司 一种改错模型训练方法、装置和文本改错方法、装置
CN106250364A (zh) * 2016-07-20 2016-12-21 科大讯飞股份有限公司 一种文本修正方法及装置
CN106326484A (zh) * 2016-08-31 2017-01-11 北京奇艺世纪科技有限公司 搜索词纠错方法及装置
US20170139774A1 (en) * 2015-11-17 2017-05-18 Kabushiki Kaisha Toshiba Correction apparatus and correction method

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103970765A (zh) * 2013-01-29 2014-08-06 腾讯科技（深圳）有限公司 一种改错模型训练方法、装置和文本改错方法、装置
US20170139774A1 (en) * 2015-11-17 2017-05-18 Kabushiki Kaisha Toshiba Correction apparatus and correction method
CN106250364A (zh) * 2016-07-20 2016-12-21 科大讯飞股份有限公司 一种文本修正方法及装置
CN106326484A (zh) * 2016-08-31 2017-01-11 北京奇艺世纪科技有限公司 搜索词纠错方法及装置

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117523590A (zh) * 2023-11-10 2024-02-06 广州方舟信息科技有限公司 一种校验生产厂商名称的方法、装置、设备及存储介质
CN117523590B (zh) * 2023-11-10 2024-05-28 广州方舟信息科技有限公司 一种校验生产厂商名称的方法、装置、设备及存储介质

Also Published As

Publication number Publication date
CN113095066A (zh) 2021-07-09

Similar Documents

Publication Publication Date Title
WO2021047286A1 (zh) 文本处理模型的训练方法、文本处理方法及装置
WO2022007823A1 (zh) 一种文本数据处理方法及装置
US20230016365A1 (en) Method and apparatus for training text classification model
WO2021233112A1 (zh) 基于多模态机器学习的翻译方法、装置、设备及存储介质
WO2020228376A1 (zh) 文本处理方法、模型训练方法和装置
CN111984766B (zh) 缺失语义补全方法及装置
CN111930942B (zh) 文本分类方法、语言模型训练方法、装置及设备
CN113239700A (zh) 改进bert的文本语义匹配设备、系统、方法及存储介质
Merkx et al. Language learning using speech to image retrieval
WO2021129411A1 (zh) 文本处理方法及装置
CN111914097A (zh) 基于注意力机制和多层级特征融合的实体抽取方法与装置
WO2023160472A1 (zh) 一种模型训练方法及相关设备
WO2022001724A1 (zh) 一种数据处理方法及装置
CN110134950B (zh) 一种字词结合的文本自动校对方法
CN109271636B (zh) 词嵌入模型的训练方法及装置
CN112101042A (zh) 文本情绪识别方法、装置、终端设备和存储介质
WO2020192523A1 (zh) 译文质量检测方法、装置、机器翻译系统和存储介质
CN113326702A (zh) 语义识别方法、装置、电子设备及存储介质
US20240152770A1 (en) Neural network search method and related device
CN114782722A (zh) 图文相似度的确定方法、装置及电子设备
WO2021129410A1 (zh) 文本处理方法及装置
CN116821307B (zh) 内容交互方法、装置、电子设备和存储介质
CN113807089A (zh) 文本数据的处理方法、神经网络的训练方法以及相关设备
CN113065350A (zh) 一种基于注意力神经网络的生物医学文本词义消岐方法
WO2023116572A1 (zh) 一种词句生成方法及相关设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20907852

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20907852

Country of ref document: EP

Kind code of ref document: A1