WO2022253138A1 - 文本处理方法、装置和电子设备 - Google Patents

文本处理方法、装置和电子设备 Download PDF

Info

Publication number
WO2022253138A1
WO2022253138A1 PCT/CN2022/095648 CN2022095648W WO2022253138A1 WO 2022253138 A1 WO2022253138 A1 WO 2022253138A1 CN 2022095648 W CN2022095648 W CN 2022095648W WO 2022253138 A1 WO2022253138 A1 WO 2022253138A1
Authority
WO
WIPO (PCT)
Prior art keywords
text
character
character string
weight
target
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2022/095648
Other languages
English (en)
French (fr)
Inventor
蒋建光
时从斌
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Vivo Mobile Communication Co Ltd
Original Assignee
Vivo Mobile Communication Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Vivo Mobile Communication Co Ltd filed Critical Vivo Mobile Communication Co Ltd
Publication of WO2022253138A1 publication Critical patent/WO2022253138A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G06F16/332—Query formulation
    • G06F16/3322—Query formulation using system suggestions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/205—Parsing
    • G06F40/216—Parsing using statistical methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/279—Recognition of textual entities
    • G06F40/289—Phrasal analysis, e.g. finite state techniques or chunking
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology

Definitions

  • the present application belongs to the technical field of communications, and in particular relates to a text processing method, device and electronic equipment.
  • the long text is usually truncated.
  • the electronic device can only extract the first N characters of the long text, and then the electronic device can only search for the N characters, so as to achieve the purpose of reducing the search time and improving the search efficiency.
  • the above search scheme may cause the electronic device to be unable to search for the information that the user needs, so that the electronic device may not be able to search for the long text.
  • the search accuracy rate is low.
  • the purpose of the embodiments of the present application is to provide a text processing method, device and electronic device, which can solve the problem of improving search efficiency and search accuracy in the process of searching long texts by electronic devices.
  • the embodiment of the present application provides a text processing method, the method includes: obtaining the first text, the first text includes M first characters, M is a positive integer; according to the characteristic information of each first character , to determine the weight of each first character, the weight of any first character is used to characterize the importance of any first character in the first text; delete the target character from the second text to obtain the target text, the first The second text is all or part of the first text, and the target character is the first character whose weight meets the first condition.
  • the embodiment of the present application provides a text processing device, which includes: an acquisition module, a determination module, and an execution module, wherein: the acquisition module is used to acquire the first text, and the first text includes M first character, M is a positive integer; the determination module is used to determine the weight of each first character according to the feature information of each first character, and the weight of any first character is used to represent the first character of any first character. The degree of importance in the text; the execution module is used to delete the target character from the second text to obtain the target text, the second text is all or part of the first text, and the target character is the first character whose weight meets the first condition character.
  • an embodiment of the present application provides an electronic device, the electronic device includes a processor, a memory, and a program or instruction stored in the memory and operable on the processor, and the program or instruction is The processor implements the steps of the method described in the first aspect when executed.
  • an embodiment of the present application provides a readable storage medium, on which a program or an instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented .
  • the embodiment of the present application provides a chip, the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions, so as to implement the first aspect the method described.
  • the text processing apparatus may determine the weight of each first character according to the feature information of each first character. Then, the text processing device can delete the target character from the second text to obtain the target text.
  • the weight of any first character is used to characterize the importance of any first character in the first text
  • the second text is all or part of the first text
  • the target character has a weight satisfying the first The first character of the condition.
  • the text processing device may delete the first characters whose weights satisfy the first condition, so that the first text may be finely simplified in units of words, and important characters in the first text may be reserved.
  • the electronic device can only search for important characters in the first text, thereby improving the search efficiency while improving the search accuracy rate.
  • FIG. 1 is a schematic flow chart of a text processing method provided in an embodiment of the present application
  • FIG. 2 is a schematic diagram of an application interface of a text processing method provided by an embodiment of the present application
  • FIG. 3 is a schematic structural diagram of a text processing device provided in an embodiment of the present application.
  • FIG. 4 is one of the structural schematic diagrams of an electronic device provided in an embodiment of the present application.
  • FIG. 5 is a second schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • first”, “second” and the like in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific sequence or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the application can be practiced in sequences other than those illustrated or described herein.
  • the objects distinguished by “first”, “second” and so on are generally one type, and the number of objects is not limited, for example, there may be one first object, or there may be multiple ones.
  • “and/or” in the specification and claims means at least one of the connected objects, and the character “/” generally means that the related objects are an "or” relationship.
  • Fig. 1 is a schematic flow chart of a text processing method provided by the embodiment of the present application, including steps 201 to 203:
  • Step 201 The text processing apparatus acquires the first text.
  • the above-mentioned first text includes M first characters, and M is a positive integer.
  • the characters in the present application may include at least one of the following: characters, numbers, letters, and symbols.
  • the first text may be input by the user through the text processing device, may also be downloaded by the text processing device, or may be obtained by the text processing device in any possible way, and this embodiment of the present application does not make any limited.
  • the text processing apparatus may acquire the first text actively, or may acquire the first text passively, which is not limited in this embodiment of the present application.
  • the user may input the first text in the search area, at this time, the text processing apparatus may acquire the first text, that is, acquire it actively.
  • the user may input the first text in the search area, and then the text processing apparatus may acquire the first text after receiving the user's first input, that is, acquire passively.
  • the above-mentioned first input may be: the user's click input on the target control, or a voice command input by the user, or a specific gesture input by the user, which may be specifically determined according to actual usage requirements.
  • the target control may be an original control or a newly added control, which is not limited in this embodiment of the present application.
  • the specific gesture in the embodiment of the present application may be any one of a click gesture, a sliding gesture, a drag gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double-press gesture, and a double-click gesture;
  • the click input can be single-click input, double-click input, or any number of click inputs, etc., and can also be long-press input or short-press input.
  • Step 202 The text processing device determines the weight of each first character according to the feature information of each first character.
  • weight of any first character is used to represent the importance of any first character in the first text.
  • the above feature information may include one piece of information, or may include multiple pieces of information, which is not limited in the embodiment of the present application.
  • the above feature information may include at least one of the following: text information, historical usage information, context information, and language probability information.
  • the above text information may include at least one of the following items: the number (or length) of any first character, the number of characters of the first character string in which any first character is located, and the number of characters of any first character. The offset in the first string where the character is located.
  • the above-mentioned historical usage information may include at least one of the following items: the number of historical clicks of any first character, and the number of occurrences of any first character in the first text.
  • the historical usage information in this application includes but not limited to the above two kinds of information.
  • the above context information may include any of the following: all or part of the characters in the first character string where any first character is located, or all or part of the characters in the first text.
  • context information in this application includes but not limited to the above two kinds of information.
  • the above-mentioned language probability information includes at least one of the following items: the language model probability of the first character string where any first character is located, the first character where any first character is randomly replaced with other characters.
  • the language model probability of the string the language model probability of the first text, the language model probability of the first text after randomly replacing any first character with other characters.
  • language probability information in this application includes but not limited to the above two kinds of information.
  • the text processing device constructs the feature vector of each first character according to the feature information of each first character, and then, the text processing device can use the word weight model to calculate the feature vector of each first character, and obtain The weight of each first character.
  • the text processing apparatus may divide the first text into sentences according to punctuation marks such as periods and commas.
  • the first text is "iPhone, apples are delicious", and the results after sentence segmentation are "iPhone” and "apples are delicious”.
  • the text processing device can perform word segmentation on each sentence after the sentence segmentation.
  • the text processing device may separate the sentence “iPhone” into “apple” and "mobile phone”. After completing word segmentation, the text processing device can obtain the following four features:
  • the first type of feature is to acquire the text feature of the first character according to the text information.
  • the text information of the word “apple” in “Apple is delicious” is a text length of 4, a word length of 2, and an offset of the word in the text of 0, that is, a vector (4, 2, 0) can be constructed, Then, after a linear transformation layer (the output is 1-dimensional), it becomes one-dimensional data, and then an activation function, and the final result is used as a text feature, which can be recorded as v1.
  • the second feature is to acquire statistical features of the first character according to historical usage information.
  • the text processing apparatus may use the value of term frequency-inverse document frequency (TF-IDF) as a statistical feature.
  • TF-IDF term frequency-inverse document frequency
  • this embodiment of the present application implements click rate weighting on the basis of the traditional TF-IDF formula.
  • the TF-IDF formula for click rate weighting in the embodiment of the present application can be expressed as:
  • tf is the number of times the word appears in the text
  • n doc is the total number of words in the document
  • m is the total number of documents in the corpus
  • df is the number of documents containing the word
  • CTR is the click-through rate of the word.
  • the traditional TF-IDF calculation can refer to the prior art, which will not be repeated here.
  • the TF-IDF value obtained from the collected text data and the user's historical click data can be recorded as v2.
  • the third feature is to obtain the word embedding (word vector) feature of the first character according to the context information.
  • the text processing device may first obtain the context-free vector representation of the word, and the text processing device searches the word vector table to obtain the context-free vector representation of each word after word segmentation.
  • the vector of "apple” is (0.1, 0.2, 0.6) and the vector of "mobile phone” is (0.2, 0.2, 0.6).
  • the vectors of "apple” in “iPhone” and “apples are delicious” are both (0.1, 0.2, 0.6), which are context-independent.
  • the text processing device can then obtain a context-dependent vector representation of the word.
  • the text processing device may input the context-independent vector representation of the word into a context-dependent encoder (for example, a transformer encoder) to obtain the context-dependent vector representation of the word.
  • a context-dependent encoder for example, a transformer encoder
  • the context-dependent vector representation of the word wi is to consider the influence of all words in the context on the word wi when calculating the representation of the word wi.
  • the context-dependent vector representation of wi can be At this time, the vector of "apple” in "iPhone” becomes (0.2, 0.3, 0.7), and the vector of "apple” in "apple is delicious” becomes (0.1, 0.4, 0.6).
  • the vector representations of the two “apples” become different due to different contexts, that is, they are context-dependent vector representations.
  • the text processing device can convert the context-dependent vector representation of the above-mentioned words into a one-dimensional feature through a linear transformation layer (the output is 1-dimensional), denoted as v3, which can be expressed as: Where w j is the j-th dimension component of the word vector, m is the dimension of the word vector, a j is the corresponding weight, and b is the bias item.
  • the fourth feature is to acquire the language model feature of the first character according to the language probability information.
  • the language model is used to evaluate the rationality of the text
  • the text processing device may use an n-gram language model or a neural network language model to evaluate the rationality of the text.
  • v4 P(w0,w,w2,...,w,wn)/P(w0,random,w2,...,random,wn), where random represents a word other than w that is randomly replaced.
  • the process of using the language model to find the probability of a text sequence first obtain the context-dependent vector representation of the word, then average the context-dependent vector representations of all words in the sentence to obtain the sentence vector representation, and then The vector representation of the sentence is transformed into one-dimensional data by a linear transformation, and then a probability value is obtained through a nonlinear activation function (such as sigmoid).
  • a nonlinear activation function such as sigmoid
  • V splicing
  • a word weight model may be used to obtain the weight score of each first character.
  • a word weight model may be used to obtain the weight score of each first character.
  • the linear transformation layer the output is 4 dimensions
  • the linear transformation The layer formula can be expressed as:
  • W is the weight matrix
  • Y is a 4-dimensional vector (corresponding to four types of scores) after linear transformation and activation function, and then adds some nonlinearity to the data through an activation function.
  • the activation function can be expressed as:
  • y j is the jth dimension of Y, corresponding to the possibility of taking the jth class score, and then after Get a probability distribution with a sum of 1, and take the score corresponding to the dimension with the highest probability as the predicted score.
  • the final probability distribution is (0.1, 0.0, 0.1, 0.8), which means that the probability of taking a score of 0 is 0.1 , the probability of taking 1 point is 0, the probability of taking 2 points is 0.1, and the probability of taking 3 points is 0.8. Therefore, the text processing device can predict that the word is 3 points.
  • the word weight prediction problem is regarded as a 0-3 four-category problem, (1, 0, 0, 0) represents 1 point, and (0, 0, 0, 1) represents 3 points , (1, 0, 0, 0) and (0, 0, 0, 1) are called the real distribution, and (0.1, 0.0, 0.1, 0.8) are called the predicted distribution, and the dimension corresponding to the largest probability in the predicted distribution is taken The score is used as the final prediction score, which is 3 points here.
  • the above-mentioned word weight model can be a logistic regression or a tree model (such as LightGBM).
  • the training of the word weight model is offline training, and the label during training is 4 grades of manually labeled word weight scores of 0-3 points.
  • the goal of the training is to make the word weight distribution predicted by the word weight model based on the input features and Human scoring is as close as possible.
  • the user can manually and subjectively judge the importance of each word in the text by collecting a certain amount of text data and user click data, and combine the user click data to score.
  • Subjective judgment is important or relatively important with high clicks, the score is 3 (that is, very important); subjective judgment is important but the click-through rate is average, the score is 2 (that is, relatively important); subjective judgment is not important and the click-through rate is low or no clicks , the score is 0 (that is, not important); the rest of the score is 1 (that is, relatively unimportant).
  • Step 203 The text processing device deletes the target character from the second text to obtain the target text.
  • the above-mentioned second text is all or part of the text in the first text
  • the above-mentioned target character is the first character whose weight satisfies the first condition.
  • the above-mentioned first condition includes any of the following: the weight is less than the first threshold, the weight is in the first threshold range, and the weight is greater than or equal to the first threshold.
  • first characters whose weights are less than a first threshold may be discarded to obtain the target text.
  • the target text is not only concise but also retains important first characters, which is more suitable for long text search requirements.
  • the text processing device after the text processing device acquires the first text including M first characters, it may determine the weight of each first character according to the characteristic information of each first character. Then, the text processing device can delete the target character from the second text to obtain the target text.
  • the weight of any first character is used to characterize the importance of any first character in the first text
  • the second text is all or part of the first text
  • the target character has a weight satisfying the first The first character of the condition.
  • the text processing device may delete the first characters whose weights satisfy the first condition, so that the first text may be finely simplified in units of words, and important characters in the first text may be reserved.
  • the electronic device can only search for important characters in the first text, thereby improving the search efficiency while improving the search accuracy rate.
  • the electronic device may first perform de-redundancy on the first text in units of character strings (that is, sentences) .
  • the method may also include the following steps 203a and 203b:
  • Step 203a The text processing device determines the weight of each first character string according to the weight of each first character.
  • the weight of any first character string is used to characterize the importance of any first character string in the first text.
  • Step 203b The text processing device deletes the target character string from the first text to obtain the second text.
  • the above-mentioned target character string is the first character string whose weight satisfies the second condition.
  • the above-mentioned second condition includes any of the following: the weight is smaller than the second threshold, the weight is in the second threshold range, and the weight is greater than or equal to the second threshold.
  • step 203a may specifically include the following step 203a1:
  • Step 203a1 The text processing device performs linear transformation and functional transformation on the weight of each first character corresponding to each first character string to determine the weight of each first character string.
  • the text processing device may use a redundant sentence remover to score each sentence.
  • the text processing device may first pass the weight of each first character through a linear transformation layer and output it as a 1-dimensional vector, the The formula of the linear transformation layer can be expressed as:
  • score(s) is the sentence score
  • score(w i ) is the word weight score of the i-th word in the sentence
  • ⁇ i is the corresponding weight.
  • the text processing device may retain sentences whose weight scores are greater than or equal to a second threshold (such as 0.5), and delete the remaining sentences to obtain the second text.
  • a second threshold such as 0.5
  • the text processing method provided by the embodiment of the present application can be applied to the scenario of improving the de-redundancy effect.
  • the user can first delete the unimportant first character string (ie sentence) in the first text, and then delete the unimportant The first character (that is, the word) is deleted from the second text in the sentence, so that the effect of de-redundancy can be improved, thereby further improving the search efficiency.
  • the method may further include the following steps 201a to 201f:
  • Step 201a The text processing device performs sentence segmentation processing on the third text to be processed to obtain at least one second character string.
  • the text processing apparatus may divide the third text into sentences according to punctuation marks such as periods and commas.
  • the third text may be input by the user through the text processing device, may also be downloaded by the text processing device, or may be obtained by the text processing device in any possible way, which is not limited in this embodiment of the present application.
  • the text processing apparatus may acquire the third text actively, or may acquire the third text passively, which is not limited in this embodiment of the present application.
  • Step 201b The text processing device performs word segmentation processing on each of the at least one second character string to obtain at least one second character.
  • the above-mentioned third text includes at least one second character string, and one second character string includes at least one second character.
  • Step 201c The text processing device acquires a vector of each second character in the at least one second character.
  • the above-mentioned vector of each second character may be understood as a context-dependent vector of each second character.
  • the process of the text processing device determining the context-dependent vector of each second character can be specifically referred to in the embodiment of the present application to determine the context-dependent vector of each first character according to the context information of each first character.
  • the description of the vector process of will not be repeated here.
  • Step 201d The text processing device determines the vector of each second character string according to the vector of each second character corresponding to each second character string.
  • the context-dependent vectors of all second characters may be input
  • the sentence encoder learns the vector representation of the entire second character string from the vector representation of each second character in any second character string. For example, summation, averaging, or weighted summation may be used.
  • the summation formula can be expressed as:
  • the formula for averaging can be expressed as:
  • the formula for weighted summation can be expressed as: Among them, v s is the vector representation of the sentence, w i is the context-dependent vector representation of the i-th word in the sentence, ⁇ i is the weight, and b is the bias item.
  • Step 201e The text processing device performs linear transformation and functional transformation on the vector of each second character string, and determines the weight of each second character string.
  • the weight of any second character string is used to represent the importance of the second character string in the third text.
  • the text processing device may input the vector representation of any second character string into a sentence classifier, wherein the sentence classifier is composed of a linear transformation layer (with a 1-dimensional output) and an activation function.
  • the text processing device can send the vector of the sentence into the linear transformation layer for linear transformation to obtain a one-dimensional value.
  • the formula of the linear transformation layer can be expressed as:
  • y s is the output of the linear transformation layer
  • w j is the j-th dimension component represented by the sentence vector
  • ⁇ j is the corresponding weight.
  • score is the score of the sentence.
  • Step 201f The text processing device deletes the second character string whose weight satisfies the third condition from the third text to obtain the first text.
  • the above-mentioned third condition includes any of the following: the weight is less than a third threshold, the weight is in the third threshold range, and the weight is greater than or equal to the third threshold.
  • the text processing device can delete sentences with a label of 0 from the third text, and select sentences with a label of 1 to form the first text, that is, perform preliminary de-redundancy on the third text , and then, the text processing device can refine the weight of each first character determined according to the text information, historical use information, context information and language probability information of each first character on the basis of the first text. redundancy. In this way, the effect of removing redundancy from text can be improved, thereby further improving the efficiency of text search.
  • a third threshold such as 0.5
  • the mobile phone screen displays a main interface 31 of a news application program, and the search box of this main interface 31 displays text 1 "country Z is a great country, why should I ask why?" So great?
  • the land is vast and rich, with 9.6 million square kilometers of land.
  • the user wants to search for the long text, the user can click the "Search" control.
  • the mobile phone can search for the The text is divided into sentences, which are divided into sentence 1 "country Z is a great country.”, sentence 2 "why is it so great?", sentence 3 "first of all, it has a vast land and resources," and sentence 4 "has a land of 9.6 million square kilometers.
  • Sentence 5 Second, it has a long history
  • Sentence 6 Up and down five thousand years.”.
  • the mobile phone can perform word segmentation for each sentence in sentence 1 to sentence 6, and determine a context-dependent vector representation of each word in sentence 1 to sentence 6.
  • the mobile phone can weight and sum the context-dependent vector representations of all words in sentence 1 to obtain the vector representation of sentence 1, and the mobile phone can perform weighted summation of the context-dependent vector representations of all words in sentence 2 to obtain sentence 2 until the mobile phone can weight and sum the context-dependent vector representations of all words in sentence 6 to obtain the vector representation of sentence 6.
  • the mobile phone can determine that the weight scores of sentence 1, sentence 3, and sentence 5 are greater than 0.5, which are labeled as label 1, and the weight scores of sentence 2, sentence 4, and sentence 6 are less than 0.5, and are labeled as label 0.
  • the mobile phone can delete sentence 2, sentence 4, and sentence 6 whose labels are 0 from text 1, and only display sentence 1, sentence 3, and sentence 5, and obtain sentence-level deredundancy Text 1. Then, the mobile phone can perform word segmentation on sentence 1, sentence 3 and sentence 5, and sentence 1 can be divided into the words "country Z", "yes", “one”, “great”, “of” and “country”. Then, the mobile phone can jointly determine the weight score of each word according to the text information, historical usage information, context information and language probability information of each word in sentence 1, where the weights of the words "is", "one” and “of” are The score is less than 1 point, and the words "Z country”, "great” and “country” have a weight score greater than 1 point. At this time, the mobile phone can delete the words “yes", “one” and "of” in sentence 1, and reserve the word “ Country Z", "Great” and "National”.
  • the text processing apparatus may execute the above step 201a to step 201f under the condition that the above feature information may include text information, historical usage information, context information and language probability information.
  • the text processing method provided by the embodiment of the present application can be applied to the scenario of improving the de-redundancy effect, and the text processing device can determine the vector of each second character string according to the vector of each second character in the third text, so that Determine the weight of each second character string according to the vector of each second character string, and delete the second character string whose weight satisfies the third condition from the third text to obtain the first text, so as to implement sentence processing on the third text Level preliminary de-redundancy, and then, the text processing device can perform word-level de-redundancy on the first text obtained by preliminary de-redundancy, so that the effect of de-redundancy on the text can be improved.
  • the text processing method provided in the embodiment of the present application may be executed by a text processing device, or a control module in the text processing device for executing the text processing method.
  • the text processing device provided in the embodiment of the present application is described by taking the text processing device executing the text processing method as an example.
  • Fig. 3 is a schematic diagram of a possible structure of a text processing device provided by the embodiment of the present application.
  • the text processing device 400 includes: an acquisition module 401, a determination module 402 and an execution module 403, wherein: , used to obtain the first text, the first text includes M first characters, and M is a positive integer; the determination module 402 is used to determine the weight of each first character according to the feature information of each first character, any The weight of a first character is used to characterize the importance of any first character in the first text; the execution module 403 is used to delete the target character from the second text to obtain the target text, and the second text is the first text All or part of the text, the target character is the first character whose weight meets the first condition.
  • the above-mentioned first text includes at least one first character string, the at least one first character string includes the above-mentioned M first characters, and one first character string includes at least one first character;
  • the determining module 402 is also used to Determine the weight of each first character string according to the weight of each first character, and the weight of any first character string is used to characterize the importance of any first character string in the first text;
  • the determining module 402 is specifically configured to perform linear transformation and functional transformation on the weight of each first character corresponding to each first character string, and determine the weight of each first character string.
  • the execution module 403 is further configured to perform sentence segmentation processing on the third text to be processed to obtain at least one second character string; and perform word segmentation processing on each second character string in the at least one second character string, obtain at least one second character; the obtaining module 401 is also used to obtain the vector of each second character in the at least one second character; the determination module 402 is also used to correspond to each second character string obtained according to the obtaining module 401 the vector of each second character, determine the vector of each second string; and carry out linear transformation and function transformation on the vector of each second character string, determine the weight of each second string, any second The weight of the string is used to characterize the importance of any second string in the third text; the execution module 403 is also used to delete the second string whose weight meets the third condition from the third text to obtain the first text.
  • the feature information includes at least one of the following: text information, historical usage information, context information, and language probability information.
  • the text processing device may determine the weight of each first character according to the characteristic information of each first character. Then, the text processing device can delete the target character from the second text to obtain the target text.
  • the weight of any first character is used to characterize the importance of any first character in the first text
  • the second text is all or part of the first text
  • the target character has a weight satisfying the first The first character of the condition.
  • the text processing device may delete the first characters whose weights satisfy the first condition, so that the first text may be finely simplified in units of words, and important characters in the first text may be reserved.
  • the electronic device can only search for important characters in the first text, thereby improving the search efficiency while improving the search accuracy rate.
  • the text processing device in this embodiment of the present application may be a device, or may be a component, an integrated circuit, or a chip in a terminal.
  • the device may be a mobile electronic device or a non-mobile electronic device.
  • the mobile electronic device may be a mobile phone, tablet computer, notebook computer, palmtop computer, vehicle electronic device, wearable device, ultra-mobile personal computer (ultra-mobile personal computer, UMPC), netbook or personal digital assistant (personal digital assistant).
  • assistant, PDA personal digital assistant
  • non-mobile electronic devices can be servers, network attached storage (Network Attached Storage, NAS), personal computer (personal computer, PC), television (television, TV), teller machine or self-service machine, etc., this application Examples are not specifically limited.
  • the text processing device in this embodiment of the present application may be a device with an operating system.
  • the operating system may be an Android (Android) operating system, an ios operating system, or other possible operating systems, which are not specifically limited in this embodiment of the present application.
  • the text processing device provided in the embodiment of the present application can realize various processes realized by the method embodiments in FIG. 1 and FIG. 2 , and details are not repeated here to avoid repetition.
  • the embodiment of the present application further provides an electronic device 500, including a processor 501, a memory 502, and programs or instructions stored in the memory 502 and operable on the processor 501,
  • an electronic device 500 including a processor 501, a memory 502, and programs or instructions stored in the memory 502 and operable on the processor 501,
  • the program or instruction is executed by the processor 501, each process of the above-mentioned text processing method embodiment can be realized, and the same technical effect can be achieved, so in order to avoid repetition, details are not repeated here.
  • the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.
  • FIG. 5 is a schematic diagram of a hardware structure of an electronic device implementing an embodiment of the present application.
  • the electronic device 100 includes but is not limited to: a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110, etc. part.
  • the electronic device 100 can also include a power supply (such as a battery) for supplying power to various components, and the power supply can be logically connected to the processor 110 through the power management system, so that the management of charging, discharging, and function can be realized through the power management system. Consumption management and other functions.
  • a power supply such as a battery
  • the structure of the electronic device shown in FIG. 5 does not constitute a limitation to the electronic device.
  • the electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange different components, and details will not be repeated here. .
  • the processor 110 is used to obtain the first text, the first text includes M first characters, and M is a positive integer; and according to the feature information of each first character, determine the weight of each first character, any The weight of a first character is used to characterize the importance of any first character in the first text; and the target character is deleted from the second text to obtain the target text, and the second text is all or part of the first text text, the target character is the first character whose weight meets the first condition.
  • the above-mentioned first text includes at least one first character string, the at least one first character string includes the above-mentioned M first characters, and one first character string includes at least one first character; the processor 110 is further configured to According to the weight of each first character, determine the weight of each first character string, the weight of any first character string is used for characterizing the importance degree of this any first character string in the first text; And from the first The target character string is deleted from the text to obtain the second text, and the target character string is the first character string whose weight satisfies the second condition.
  • the processor 110 is specifically configured to perform linear transformation and functional transformation on the weight of each first character corresponding to each first character string to determine the weight of each first character string.
  • the processor 110 is further configured to perform sentence segmentation processing on the third text to be processed to obtain at least one second character string; perform word segmentation processing on each second character string in the at least one second character string to obtain at least one second character; obtaining the vector of each second character in the at least one second character; determining the vector of each second character string according to the vector of each second character corresponding to each second character string; Perform linear transformation and function transformation on the vector of each second character string to determine the weight of each second character string, and the weight of any second character string is used to represent the importance of any second character string in the third text degree; and deleting the second character string whose weight satisfies the third condition from the third text to obtain the first text.
  • the feature information includes at least one of the following: text information, historical usage information, context information, and language probability information.
  • the electronic device after acquiring the first text including M first characters, the electronic device may determine the weight of each first character according to feature information of each first character. Then, the electronic device can delete the target character from the second text to obtain the target text.
  • the weight of any first character is used to characterize the importance of any first character in the first text
  • the second text is all or part of the first text
  • the target character has a weight satisfying the first The first character of the condition.
  • the electronic device can delete the first characters whose weights satisfy the first condition, so that the first text can be finely simplified in units of words, and important characters in the first text can be reserved. In this way, in the process of searching a long text, the electronic device can only search for important characters in the first text, thereby improving the search efficiency while improving the search accuracy rate.
  • the input unit 104 may include a graphics processor (Graphics Processing Unit, GPU) 1041 and a microphone 1042, and the graphics processor 1041 is used for the image capture device (such as the image data of the still picture or video obtained by the camera) for processing.
  • the display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like.
  • the user input unit 107 includes a touch panel 1071 and other input devices 1072 .
  • the touch panel 1071 is also called a touch screen.
  • the touch panel 1071 may include two parts, a touch detection device and a touch controller.
  • Other input devices 1072 may include, but are not limited to, physical keyboards, function keys (such as volume control keys, switch keys, etc.), trackballs, mice, and joysticks, which will not be repeated here.
  • Memory 109 may be used to store software programs as well as various data, including but not limited to application programs and operating systems.
  • the processor 110 may integrate an application processor and a modem processor, wherein the application processor mainly processes operating systems, user interfaces, and application programs, and the modem processor mainly processes wireless communications. It can be understood that the foregoing modem processor may not be integrated into the processor 110 .
  • the embodiment of the present application also provides a readable storage medium, on which a program or an instruction is stored, and when the program or instruction is executed by a processor, each process of the above-mentioned text processing method embodiment is realized, and the same To avoid repetition, the technical effects will not be repeated here.
  • the processor is the processor in the electronic device described in the above embodiments.
  • the readable storage medium includes computer readable storage medium, such as computer read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk, etc.
  • the embodiment of the present application further provides a chip, the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the above text processing method embodiment Each process can achieve the same technical effect, so in order to avoid repetition, it will not be repeated here.
  • chips mentioned in the embodiments of the present application may also be called system-on-chip, system-on-chip, system-on-a-chip, or system-on-a-chip.
  • the term “comprising”, “comprising” or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or apparatus comprising a set of elements includes not only those elements, It also includes other elements not expressly listed, or elements inherent in the process, method, article, or device. Without further limitations, an element defined by the phrase “comprising a " does not preclude the presence of additional identical elements in the process, method, article, or apparatus comprising that element.
  • the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved. Functions are performed, for example, the described methods may be performed in an order different from that described, and various steps may also be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • General Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Mathematical Physics (AREA)
  • Data Mining & Analysis (AREA)
  • Probability & Statistics with Applications (AREA)
  • Databases & Information Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Evolutionary Computation (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

提供了一种文本处理方法、装置和电子设备，属于通信技术领域。方法包括：获取第一文本，第一文本包括M个第一字符，M为正整数；根据每个第一字符的特征信息，确定每个第一字符的权重，任一第一字符的权重用于表征任一第一字符在第一文本中的重要程度；从第二文本中删除目标字符，得到目标文本，第二文本为第一文本的全部或部分文本，目标字符为权重满足第一条件的第一字符。

Description

文本处理方法、装置和电子设备
相关申请的交叉引用
本申请主张在2021年05月31日在中国提交的中国专利申请号202110603987.6的优先权，其全部内容通过引用包含于此。
技术领域
本申请属于通信技术领域，具体涉及一种文本处理方法、装置和电子设备。
背景技术
随着电子设备具有的搜索功能越来越强大，用户可以使用电子设备的搜索功能对长文本进行搜索。但是由于长文本内容过多，可能会导致电子设备搜索过程耗时过长、搜索效率低。
目前，为了提高搜索效率，减少搜索时间等，用户在使用电子设备的搜索功能对长文本进行搜索的过程中，通常是通过截断长文本来实现的。具体地，电子设备可以仅提取长本文的前N个字符，然后，电子设备可以只对该N个字符进行搜索，如此，达到减少搜索时长，提高搜索效率的目的。
然而，由于未被搜索的文本中可能包括用户需求搜索的信息，因此，通过上述的搜索方案可能会导致电子设备无法搜索到用户需求的信息，从而使得电子设备在对长文本进行搜索的过程中搜索正确率低。
发明内容
本申请实施例的目的是提供一种文本处理方法、装置和电子设备，能够解决在电子设备对长文本进行搜索的过程中，能够在提高搜索效率的同时提高搜索正确率的问题。
第一方面，本申请实施例提供了一种文本处理方法，该方法包括：获取第一文本，该第一文本包括M个第一字符，M为正整数；根据每个第一字符的特征信息，确定每个第一字符的权重，任一第一字符的权重用于表征该任一第一字符在第一文本中的重要程度；从第二文本中删除目标字符，得到目标文本，该第二文本为该第一文本的全部或部分文本，该目标字符为权重满足第一条件的第一字符。
第二方面，本申请实施例提供了一种文本处理装置，该装置包括：获取模块、确定模块和执行模块，其中：获取模块，用于获取第一文本，该第一文本包括M个第一字符，M为正整数；确定模块，用于根据每个第一字符的特征信息，确定每个第一字符的权重，任一第一字符的权重用于表征该任一第一字符在第一文本中的重要程度；执行模块，用于从第二文本中删除目标字符，得到目标文本，该第二文本为第一文本的全部或部分文本，该目标字符为权重满足第一条件的第一字符。
第三方面，本申请实施例提供了一种电子设备，该电子设备包括处理器、存储器及存储在所述存储器上并可在所述处理器上运行的程序或指令，所述程序或指令被所述处理器执行时实现如第一方面所述的方法的步骤。
第四方面，本申请实施例提供了一种可读存储介质，所述可读存储介质上存储程序或指令，所述程序或指令被处理器执行时实现如第一方面所述的方法的步骤。
第五方面，本申请实施例提供了一种芯片，所述芯片包括处理器和通信接口，所述通信接口和所述处理器耦合，所述处理器用于运行程序或指令，实现如第一方面所述的方法。
在本申请实施例中，文本处理装置在获取包括M个第一字符的第一文本之后，可以根据每个第一字符的特征信息，确定每个第一字符的权重。然后，文本处理装置可以从第二文本中删除目标字符，得到目标文本。其中，任一第一字符的权重用于表征该任一第一字符在第一文本中的重要程度，该第二文本为该第一文本的全部或部分文本，该目标字符为权重满足第一条件的第一字符。通过上述方案，在电子设备对长文本(如第一文本)进行搜索的情况下，文本处理装置可以根据第一文本中每个第一字符的特征信息，确定每个第一字符的权重。然后，文本处理装置可以删除权重满足第一条件的第一字符，从而可以以词为单位精细地简化第一文本，保留第一文本中的重要字符。如此，使得电子设备在对长文本进行搜索的过程中，可以仅搜索第一文本中的重要字符，进而能够在提高搜索效率的同时提高搜索正确率。
附图说明
图1为本申请实施例提供的一种文本处理方法流程示意图；
图2为本申请实施例提供的一种文本处理方法应用的界面示意图；
图3为本申请实施例提供的一种文本处理装置的结构示意图；
图4为本申请实施例提供的一种电子设备的结构示意图之一；
图5为本申请实施例提供的一种电子设备的结构示意图之二。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚地描述，显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员获得的所有其他实施例，都属于本申请保护的范围。
本申请的说明书和权利要求书中的术语“第一”、“第二”等是用于区别类似的对象，而不用于描述特定的顺序或先后次序。应该理解这样使用的数据在适当情况下可以互换，以便本申请的实施例能够以除了在这里图示或描述的那些以外的顺序实施。且“第一”、“第二”等所区分的对象通常为一类，并不限定对象的个数，例如第一对象可以是一个，也可以是多个。此外，说明书以及权利要求中“和/或”表示所连接对象的至少其中之一，字符“/”，一般表示前后关联对象是一种“或”的关系。
下面结合附图，通过具体的实施例及其应用场景对本申请实施例提供的文本处理方法进行详细地说明。
图1为本申请实施例提供的一种文本处理方法流程示意图，包括步骤201至步骤203：
步骤201：文本处理装置获取第一文本。
其中，上述第一文本包括M个第一字符，M为正整数。
在本申请实施例中，本申请中的字符可以包括以下至少一项：文字，数字，字母，符号。
在本申请实施例中，第一文本可以是用户通过文本处理装置输入的，也可以是文本处理装置下载的，还可以是文本处理装置通过任意可能的方式获取的，本申请实施例对此不作限定。
在本申请实施例中，文本处理装置可以主动获取第一文本，也可以被动获取第一文本，本申请实施例对此不作限定。
在一种示例中，用户可以在搜索区域中输入第一文本，此时，文本处理装置可以获取该第一文本，即主动获取。
在另一种示例中，用户可以在搜索区域中输入第一文本，然后，文本处理装置可以在接收到用户的第一输入之后，获取该第一文本，即被动获取。
示例性地，上述的第一输入可以为：用户对目标控件的点击输入，或者为用户输入的语音指令，或者为用户输入的特定手势，具体的可以根据实际使用需求确定，本申请实施例对此不作限定。其中，该目标控件可以为原有控件，也可以为新增控件，本申请实施例对此不作限定。
本申请实施例中的特定手势可以为单击手势、滑动手势、拖动手势、压力识别手势、长按手势、面积变化手势、双按手势、双击手势中的任意一种；本申请实施例中的点击输入可以为单击输入、双击输入或任意次数的点击输入等，还可以为长按输入或短按输入。
步骤202：文本处理装置根据每个第一字符的特征信息，确定每个第一字符的权重。
其中，任一第一字符的权重用于表征该任一第一字符在第一文本中的重要程度。
在本申请实施例中，上述的特征信息可以包括一个信息，也可以包括多个信息，本申请实施例对此不作限定。
可选地，在本申请实施例中，上述的特征信息可以包括以下至少一项：文本信息，历史使用信息，上下文信息，语言概率信息。
示例性地，上述的文本信息可以包括以下至少一项：任一第一字符字符数量(或称为长度)，该任一第一字符所在的第一字符串的字符数量，该任一第一字符在所在的第一字符串中的偏移量。
需要说明的是，本申请中的本文信息包括但不限于上述的三种信息。
示例性地，上述的历史使用信息可以包括以下至少一项：任一第一字符的历史点击次数，该任一第一字符在第一文本出现的次数。
需要说明的是，本申请中的历史使用信息包括但不限于上述的两种信息。
示例性地，上述的上下文信息可以包括以下任一项：任一第一字符所在的第一字符串中的全部或部分字符，第一文本中的全部或部分字符。
需要说明的是，本申请中的上下文信息包括但不限于上述的两种信息。
示例性的，上述的语言概率信息包括以下至少一项：任一第一字符所在的第一字符串的语言模型概率，将该任一第一字符随机替换为其他字符后的所在的第一字符串的语言模型概率，第一文本的语言模型概率，将该任一第一字符随机替换为其他字符后的第一文本的语言模型概率。
需要说明的是，本申请中的语言概率信息包括但不限于上述的两种信息。
示例性地，文本处理装置根据每个第一字符的特征信息，构建每个第一字 符的特征向量，然后，文本处理装置可以采用词权重模型对每个第一字符的特征向量进行计算，得到每个第一字符的权重。
具体的构建字符的特征向量的过程可以如下：
示例性地，文本处理装置可以根据句号和逗号等标点符号对第一文本进行分句。例如，第一文本为“苹果手机，苹果很好吃”，分句后的结果为“苹果手机”和“苹果很好吃”。然后，文本处理装置可以对分句后的各个句子进行分词。例如，文本处理装置可以将句子“苹果手机”分成“苹果”和“手机”。在完成分词之后，文本处理装置可以获取如下四种特征：
第一种特征，根据文本信息获取第一字符的文本特征。
例如，“苹果很好吃”中的“苹果”一词的文本信息为文本长度4，词的长度2，词在文本中的偏移量0，即可以构建向量(4，2，0)，然后，经过一个线性变换层(输出为1维)将其变成一维的数据，再经过一个激活函数，将最后的结果作为文本特征，可以记为v1。
第二种特征，根据历史使用信息获取第一字符的统计特征。
在一种示例中，文本处理装置可以将词频逆文本频率指数(term frequency–inverse document frequency，TF-IDF)的值作为统计特征。
其中，传统的TF-IDF公式可以表示为：
Figure PCTCN2022095648-appb-000001
示例性地，本申请实施例在传统的TF-IDF公式的基础上进行了点击率加权。具体地，本申请实施例中的进行点击率加权的TF-IDF公式可以表示为：
Figure PCTCN2022095648-appb-000002
其中，tf为词在文本中出现的次数，n doc为文档的总词数，m为语料库的总文档数，df为包含该词的文档数，CTR为该词的点击率。
需要说明的是，传统的TF-IDF计算可以参考现有技术，此处不再赘述。本申请实施例中从收集的文本数据和用户历史点击数据中统计得到的TF-IDF值，可以记为v2。
第三种特征，根据上下文信息获取第一字符的词嵌入(词向量)特征。
示例性地，文本处理装置可以先获得词的上下文无关的向量表示，文本处理装置在词向量表中查寻，获得分词后每个词的词的上下文无关的向量表示。例如，查询得到“苹果”的向量为(0.1，0.2，0.6)，“手机”的向量为(0.2，0.2，0.6)。需要说明的是，此时“苹果手机”和“苹果很好吃”中的“苹果” 的向量均为(0.1，0.2，0.6)，均与上下文无关。然后，文本处理装置可以获得词的上下文相关的向量表示。文本处理装置可以将上述词的上下文无关的向量表示输入上下文相关的编码器(例如，transformer编码器)，获得词的上下文相关的向量表示。
举例说明，假设一段文本包含n个词，s＝(w1,w2,…,wn)，其中wi表示第i个词，vi表示第i个词的上下文无关的向量表示。而词wi的上下文相关的向量表示就是在计算词wi的表示时考虑上下文所有词对词wi的影响。具体地，上下文所有词wj对wi的影响程度为点乘相似性aij＝vi·vj，则wi的上下文相关的向量表示可以为
Figure PCTCN2022095648-appb-000003
此时“苹果手机”中的“苹果”的向量变为(0.2，0.3，0.7)，而“苹果很好吃”中的“苹果”的向量变为(0.1，0.4，0.6)。如此，两个“苹果”的向量表示由于上下文不同而变得不同了，即为上下文相关的向量表示。
示例性地，文本处理装置可以将上述词的上下文相关的向量表示经过一个线性变换层(输出为1维)变成一维特征，记为v3，可以表示为：
Figure PCTCN2022095648-appb-000004
其中w j为词向量的第j维分量，m为词向量的维数，a j为相应的权重，b为偏置项。
第四种特征，根据语言概率信息获取第一字符的语言模型特征。
示例性地，语言模型用于评估文本的合理性，例如，文本处理装置可以采用n-gram语言模型或神经网络语言模型对文本的合理性进行评估。本申请实施例中的词w的语言模型特征可以记为v4，定义为v4＝输入文本的语言模型概率/输入文本将该词随机替换为别的词后的语言模型概率，即可以表示为：
v4＝P(w0,w,w2,…,w,wn)/P(w0,random,w2,…,random,wn)，其中，random表示随机替换的w以外的词。
在一种示例中，采用语言模型求一段文本序列的概率的过程：首先获得词的上下文相关的向量表示，然后对句子中的所有词的上下文相关的向量求平均得到句子的向量表示，然后对句子的向量表示做一个线性变换变成一维数据，再经过一个非线性激活函数(比如sigmoid)得到一个概率值。
示例性地，在文本处理装置获得上述四类特征之后，可以拼接获得词的特征向量表示，记为V＝(v1，v2，v3，v4)。如此，文本处理装置可以获得第一文本中每个第一字符的特征向量。
示例性地，在文本处理装置获得第一文本中每个第一字符的特征向量之后，可以采用词权重模型获得每个第一字符的权重得分。具体地，以上述获得的词的特征向量V作为输入为例，先经过一个线性变换层(输出为4维)将数 据变成4维Y＝(y1，y2，y3，y4)，该线性变换层的公式可以表示为：
Y＝W*V+B     (3)
其中，W为权重矩阵，Y为经过线性变换和激活函数的4维向量(对应四类分值)，再经过一个激活函数给数据增加一些非线性，该激活函数可以表示为：
Y＝sigmoid(Y)      (4)
最后经过一个softmax分类器得到预测词权重得分，公式如下：
Figure PCTCN2022095648-appb-000005
其中，y j为Y的第j个维度，对应取第j类分值的可能性大小，然后经过
Figure PCTCN2022095648-appb-000006
得到总和为1的概率分布，取其中概率最大的维度对应的分值作为预测得分，假设最后得到的概率分布为(0.1，0.0，0.1，0.8)，该概率分布表示取0分的概率为0.1，取1分的概率为0，取2分的概率为0.1，取3分的概率为0.8，因此，文本处理装置可以预测该词为3分。
需要说明的是，本申请实施例将词权重预测问题看做一个0-3的四分类问题，(1，0，0，0)表示1分，(0，0，0，1)表示3分，(1，0，0，0)和(0，0，0，1)称为真实分布，而(0.1，0.0，0.1，0.8)称为预测分布，取预测分布中概率最大的维度对应的分值作为最终的预测得分，此处即为3分。
示例性地，上述的词权重模型可以选择逻辑回归或树模型(如LightGBM)。
示例性地，词权重模型的训练为离线训练，训练时的标签为0-3分的4档人工标注的词权重得分，训练的目标为使得该词权重模型基于输入特征预测的词权重分布与人工评分尽可能的接近。
需要说明的是，用户可以通过收集到的一定数量的文本数据及用户点击数据，人工主观判断文本中每个词的重要性并结合用户点击数据进行打分。具体的：主观判定重要或比较重要且高点击，得分为3(即非常重要)；主观判定重要但点击率一般，得分为2(即比较重要)；主观判定不重要且点击率低或无点击，得分为0(即不重要)；其余的得分为1(即比较不重要)。
步骤203：文本处理装置从第二文本中删除目标字符，得到目标文本。
其中，上述第二文本为第一文本中的全部或部分文本，上述目标字符为权重满足第一条件的第一字符。
在本申请实施例中，上述的第一条件包括以下任一项：权重小于第一阈值， 权重处于第一阈值区间，权重大于或等于该第一阈值。
示例性地，在文本处理装置获得第一文本中每个第一字符的权重之后，可以丢弃掉权重小于第一阈值(比如1分)的第一字符，获得目标文本。该目标文本不但简洁且保留了重要的第一字符，更加适合长文本搜索的需求。
本申请实施例提供的文本处理方法，文本处理装置在获取包括M个第一字符的第一文本之后，可以根据每个第一字符的特征信息，确定每个第一字符的权重。然后，文本处理装置可以从第二文本中删除目标字符，得到目标文本。其中，任一第一字符的权重用于表征该任一第一字符在第一文本中的重要程度，该第二文本为该第一文本的全部或部分文本，该目标字符为权重满足第一条件的第一字符。通过上述方案，在电子设备对长文本(如第一文本)进行搜索的情况下，文本处理装置可以根据第一文本中每个第一字符的特征信息，确定每个第一字符的权重。然后，文本处理装置可以删除权重满足第一条件的第一字符，从而可以以词为单位精细地简化第一文本，保留第一文本中的重要字符。如此，使得电子设备在对长文本进行搜索的过程中，可以仅搜索第一文本中的重要字符，进而能够在提高搜索效率的同时提高搜索正确率。
可选地，在本申请实施例中，电子设备在以字符为单位精细化的对第一文本进行去冗余之前，可以先以字符串(即句子)为单位对第一文本进行去冗余。
示例性地，在第一文本包括至少一个第一字符串，该至少一个第一字符串包括上述M个第一字符，一个第一字符串包括至少一个第一字符的情况下，在上述的步骤203之前，该方法还可以包括如下步骤203a和步骤203b：
步骤203a：文本处理装置根据上述每个第一字符的权重，确定每个第一字符串的权重。
其中，任一第一字符串的权重用于表征该任一第一字符串在第一文本中的重要程度。
步骤203b：文本处理装置从第一文本中删除目标字符串，得到第二文本。
其中，上述目标字符串为权重满足第二条件的第一字符串。
示例性地，上述的第二条件包括以下任一项：权重小于第二阈值，权重处于第二阈值区间，权重大于或等于该第二阈值。
可选地，在本申请实施例中，上述的步骤203a具体可以包括如下步骤203a1：
步骤203a1：文本处理装置将每个第一字符串对应的每个第一字符的权重进行线性变换和函数变换，确定每个第一字符串的权重。
示例性地，文本处理装置可以采用一个冗余句子剔除器对各个句子进行打分，具体的，文本处理装置可以将每个第一字符的权重先经过一个线性变换层将输出为1维向量，该线性变换层的公式可以表示为：
Figure PCTCN2022095648-appb-000007
其中，score(s)为句子得分，score(w i)为句子中第i个词的词权重得分，α i为相应的权重。再经过sigmoid激活函数将数据变成0-1之间，并以0-1之间的数据作为句子的权重得分，该激活函数的公式可以表示为：
score(s)＝sigmoid(score(s))   (7)
最后，文本处理装置可以保留权重得分大于或等于第二阈值(如0.5)的句子，删除其余的句子得到第二文本。
本申请实施例提供的文本处理方法可以应用于提高去冗余效果的场景中，用户可以先删除第一文本中不重要的第一字符串(即句子)，然后，再从删除了不重要的句子中的第二文本中再删除第一字符(即词)，如此可以提高去冗余的效果，从而进一步提高搜索效率。
可选地，在本申请实施例中，在上述的步骤201之前，该方法还可以包括如下步骤201a至步骤201f：
步骤201a：文本处理装置对待处理的第三文本执行分句处理，得到至少一个第二字符串。
示例性地，文本处理装置可以根据句号和逗号等标点符号对第三文本进行分句。
示例性的，第三文本可以是用户通过文本处理装置输入的，也可以是文本处理装置下载的，还可以是文本处理装置通过任意可能的方式获取的，本申请实施例对此不作限定。
示例性的，文本处理装置可以主动获取第三文本，也可以被动获取第三文本，本申请实施例对此不作限定。
步骤201b：文本处理装置对上述至少一个第二字符串中的每个第二字符串执行分词处理，得到至少一个第二字符。
可以理解，上述第三文本包括至少一个第二字符串，一个第二字符串包括至少一个第二字符。
需要说明的是，上述的分句处理和分词处理的具体过程可以参考相关技术，此处不再赘述。
步骤201c：文本处理装置获取上述至少一个第二字符中的每个第二字符的 向量。
示例性的，上述每个第二字符的向量可以理解为每个第二字符的上下文相关向量。
需要说明的是，文本处理装置确定每个第二字符的上下文相关的向量的过程，具体可以参考本申请实施例中对根据每个第一字符的上下文信息，确定每个第一字符的上下文相关的向量过程的描述，此处不再赘述。
步骤201d：文本处理装置根据每个第二字符串对应的每个第二字符的向量，确定每个第二字符串的向量。
示例性地，针对任一第二字符串，在文本处理装置确定该任一第二字符串中的每个第二字符的上下文相关的向量之后，可以将所有第二字符的上下文相关的向量输入一个句子编码器，句子编码器从该任一第二字符串中各个第二字符的向量表示学习到整个第二字符串的向量表示。例如，可以采用求和、求平均或者加权求和的方式。其中，求和的公式可以表示为：
Figure PCTCN2022095648-appb-000008
求平均的公式可以表示为：
Figure PCTCN2022095648-appb-000009
加权求和的公式可以表示为：
Figure PCTCN2022095648-appb-000010
其中，v s为句子的向量表示，w i为句子中第i个词的上下文相关的向量表示，α i为权重，b为偏置项。
步骤201e：文本处理装置将每个第二字符串的向量进行线性变换和函数变换，确定每个第二字符串的权重。
其中，任一第二字符串的权重用于表征该任一第二字符串在第三文本中的重要程度。
示例性地，文本处理装置可以将任一第二字符串的向量表示输入一个句子分类器，其中，该句子分类器由一个线性变换层(输出为1维)和一个激活函数组成。具体的，文本处理装置可以把句子的向量送入线性变换层进行线性变换，得到一维的数值，该线性变换层的公式可以表示为：
Figure PCTCN2022095648-appb-000011
其中，y s为线性变换层的输出，w j为句子向量表示的第j维分量，α j为相应的权重。然后，文本处理装置可以把一维的数值送入激活函数进行变换，获得一个0-1的数值作为句子的得分score。该激活函数的公式可以表示为：
score＝sigmoid(y s)   (9)
其中，score为句子的得分。
步骤201f：文本处理装置从第三文本中删除权重满足第三条件的第二字符串，得到第一文本。
示例性地，上述的第三条件包括以下任一项：权重小于第三阈值，权重处于第三阈值区间，权重大于或等于该第三阈值。
在一种示例中，在第二字符串的得分(score)在0-1之间时，当score大于或等于第三阈值(如0.5)时预测为标签1，否则预测为标签0，其中，1表示属于摘要，0表示不属于摘要，因此，文本处理装置可以从第三文本中删除标签为0的句子，选择标签为1的句子组成第一文本，即对第三文本进行初步去冗余，然后，文本处理装置可以在第一文本的基础上，再根据每个第一字符的文本信息、历史使用信息、上下文信息和语言概率信息确定的每个第一字符的权重进行精细化的去冗余。如此，可以提高对文本去冗余的效果，从而进一步提高文本搜索的效率。
举例说明，如图2中的(a)所示，手机屏幕显示有新闻应用程序的主界面31，该主界面31的搜索框中显示有文本1“Z国是一个伟大的国家，要问为何如此伟大？首先地大物博，拥有960万平方公里的土地。其次历史悠久，上下五千年。”，当用户想要搜索该长文本时，用户可以点击“搜索”控件，此时，手机可以对该文本进行分句，分为句子1“Z国是一个伟大的国家。”、句子2“要问为何如此伟大？”、句子3“首先地大物博，”、句子4“拥有960万平方公里的土地。”、句5“其次历史悠久，”和句子6“上下五千年。”。
然后，手机可以对句子1至句子6中的每个句子进行分词，并确定句子1至句子6中每个词的上下文相关的向量表示。接着，手机可以对句子1中的所有词的上下文相关的向量表示加权求和得到该句子1的向量表示，手机可以对句子2中的所有词的上下文相关的向量表示加权求和得到该句子2的向量表示，直至手机可以对句子6中的所有词的上下文相关的向量表示加权求和得到该句子6的向量表示。然后，手机可以根据句子的向量表示确定句子1、句子3和句子5的权重得分大于0.5，标记为标签1，句子2、句子4和句子6的权重得分小于0.5，标记为标签0。
接着，如图2中(b)所示，手机可以从文本1中删除标签为0的句子2、句子4和句子6，仅显示句子1、句子3和句子5，得到句子级去冗余后的文本1。然后，手机可以对句子1、句子3和句子5进行分词，句子1可以分成词“Z国”、“是”、“一个”、“伟大”、“的”和“国家”。然后，手机可以根据句子1中的每个词的文本信息、历史使用信息、上下文信息和语言概率信息联合确定 每个词的权重得分，其中，词“是”、“一个”和“的”权重得分小于1分，词“Z国”、“伟大”和“国家”权重得分大于1分，此时，手机可以删除句子1中的词“是”、“一个”和“的”，保留词“Z国”、“伟大”和“国家”。
最后，以此类推，对句子2和句子3进行处理后，如图2中的(c)所示，手机显示词“Z国”、“伟大”、“国家”、“地大物博”和“历史悠久”。
需要说明的是，手机显示句子1、句子3和句子5以及手机显示词“Z国”、“伟大”、“国家”、“地大物博”和“历史悠久”，只是为了方便理解本申请，在实际去冗余的过程中，手机可以不执行显示句子1、句子3和句子5以及显示词“Z国”、“伟大”、“国家”、“地大物博”和“历史悠久”的步骤。
在一种示例中，文本处理装置可以在上述的特征信息可以包括文本信息、历史使用信息、上下文信息和语言概率信息的情况下，执行上步骤201a至步骤201f。
本申请实施例提供的文本处理方法可以应用于提高去冗余效果的场景中，文本处理装置可以根据第三文本中的每个第二字符的向量，确定每个第二字符串的向量，从而根据每个第二字符串的向量确定每个第二字符串的权重，并从第三文本中删除权重满足第三条件的第二字符串，得到第一文本，从而实现对第三文本进行句子级的初步去冗余，然后，文本处理装置可以在对初步去冗余得到的第一文本进行词级去冗余，如此，可以提高对文本去冗余的效果。
需要说明的是，本申请实施例提供的文本处理方法，执行主体可以为文本处理装置，或者该文本处理装置中的用于执行文本处理方法的控制模块。本申请实施例中以文本处理装置执行文本处理方法为例，说明本申请实施例提供的文本处理装置。
图3为实现本申请实施例提供的一种文本处理装置的可能的结构示意图，如图3所示，文本处理装置400包括：获取模块401、确定模块402和执行模块403，其中：获取模块401，用于获取第一文本，该第一文本包括M个第一字符，M为正整数；确定模块402，用于根据每个第一字符的特征信息，确定每个第一字符的权重，任一第一字符的权重用于表征该任一第一字符在第一文本中的重要程度；执行模块403，用于从第二文本中删除目标字符，得到目标文本，该第二文本为第一文本的全部或部分文本，该目标字符为权重满足第一条件的第一字符。
可选地，上述第一文本包括至少一个第一字符串，该至少一个第一字符串包括上述M个第一字符，一个第一字符串包括至少一个第一字符；确定模块 402，还用于根据每个第一字符的权重，确定每个第一字符串的权重，任一第一字符串的权重用于表征该任一第一字符串在第一文本中的重要程度；执行模块403，还用于从第一文本中删除目标字符串，得到第二文本，该目标字符串为权重满足第二条件的第一字符串。
可选地，确定模块402，具体用于将每个第一字符串对应的每个第一字符的权重进行线性变换和函数变换，确定每个第一字符串的权重。
可选地，执行模块403，还用于对待处理的第三文本执行分句处理，得到至少一个第二字符串；以及对至少一个第二字符串中的每个第二字符串执行分词处理，得到至少一个第二字符；获取模块401，还用于获取至少一个第二字符中的每个第二字符的向量；确定模块402，还用于根据获取模块401获取的每个第二字符串对应的每个第二字符的向量，确定每个第二字符串的向量；以及将每个第二字符串的向量进行线性变换和函数变换，确定每个第二字符串的权重，任一第二字符串的权重用于表征该任一第二字符串在第三文本中的重要程度；执行模块403，还用于从第三文本中删除权重满足第三条件的第二字符串，得到第一文本。
可选地，特征信息包括以下至少一项：文本信息，历史使用信息，上下文信息，语言概率信息。
本申请实施例提供的文本处理装置，文本处理装置在获取包括M个第一字符的第一文本之后，可以根据每个第一字符的特征信息，确定每个第一字符的权重。然后，文本处理装置可以从第二文本中删除目标字符，得到目标文本。其中，任一第一字符的权重用于表征该任一第一字符在第一文本中的重要程度，该第二文本为该第一文本的全部或部分文本，该目标字符为权重满足第一条件的第一字符。通过上述方案，在电子设备对长文本(如第一文本)进行搜索的情况下，文本处理装置可以根据第一文本中每个第一字符的特征信息，确定每个第一字符的权重。然后，文本处理装置可以删除权重满足第一条件的第一字符，从而可以以词为单位精细地简化第一文本，保留第一文本中的重要字符。如此，使得电子设备在对长文本进行搜索的过程中，可以仅搜索第一文本中的重要字符，进而能够在提高搜索效率的同时提高搜索正确率。
本实施例中各种实现方式具有的有益效果具体可以参见上述方法实施例中相应实现方式所具有的有益效果，为避免重复，此处不再赘述。
本申请实施例中的文本处理装置可以是装置，也可以是终端中的部件、集成电路、或芯片。该装置可以是移动电子设备，也可以为非移动电子设备。示 例性地，移动电子设备可以为手机、平板电脑、笔记本电脑、掌上电脑、车载电子设备、可穿戴设备、超级移动个人计算机(ultra-mobile personal computer，UMPC)、上网本或者个人数字助理(personal digital assistant，PDA)等，非移动电子设备可以为服务器、网络附属存储器(Network Attached Storage，NAS)、个人计算机(personal computer，PC)、电视机(television，TV)、柜员机或者自助机等，本申请实施例不作具体限定。
本申请实施例中的文本处理装置可以为具有操作系统的装置。该操作系统可以为安卓(Android)操作系统，可以为ios操作系统，还可以为其他可能的操作系统，本申请实施例不作具体限定。
本申请实施例提供的文本处理装置能够实现图1和图2的方法实施例实现的各个过程，为避免重复，这里不再赘述。
可选的，如图4所示，本申请实施例还提供一种电子设备500，包括处理器501，存储器502，存储在存储器502上并可在所述处理器501上运行的程序或指令，该程序或指令被处理器501执行时实现上述文本处理方法实施例的各个过程，且能达到相同的技术效果，为避免重复，这里不再赘述。
需要注意的是，本申请实施例中的电子设备包括上述所述的移动电子设备和非移动电子设备。
图5为实现本申请实施例的一种电子设备的硬件结构示意图。
该电子设备100包括但不限于：射频单元101、网络模块102、音频输出单元103、输入单元104、传感器105、显示单元106、用户输入单元107、接口单元108、存储器109、以及处理器110等部件。
本领域技术人员可以理解，电子设备100还可以包括给各个部件供电的电源(比如电池)，电源可以通过电源管理系统与处理器110逻辑相连，从而通过电源管理系统实现管理充电、放电、以及功耗管理等功能。图5中示出的电子设备结构并不构成对电子设备的限定，电子设备可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件布置，在此不再赘述。
其中，处理器110，用于获取第一文本，该第一文本包括M个第一字符，M为正整数；并根据每个第一字符的特征信息，确定每个第一字符的权重，任一第一字符的权重用于表征该任一第一字符在第一文本中的重要程度；以及从第二文本中删除目标字符，得到目标文本，该第二文本为第一文本的全部或部分文本，该目标字符为权重满足第一条件的第一字符。
可选地，上述第一文本包括至少一个第一字符串，该至少一个第一字符串 包括上述M个第一字符，一个第一字符串包括至少一个第一字符；处理器110，还用于根据每个第一字符的权重，确定每个第一字符串的权重，任一第一字符串的权重用于表征该任一第一字符串在第一文本中的重要程度；以及从第一文本中删除目标字符串，得到第二文本，该目标字符串为权重满足第二条件的第一字符串。
可选地，处理器110，具体用于将每个第一字符串对应的每个第一字符的权重进行线性变换和函数变换，确定每个第一字符串的权重。
可选地，处理器110，还用于对待处理的第三文本执行分句处理，得到至少一个第二字符串；对至少一个第二字符串中的每个第二字符串执行分词处理，得到至少一个第二字符；获取至少一个第二字符中的每个第二字符的向量；根据每个第二字符串对应的每个第二字符的向量，确定每个第二字符串的向量；将每个第二字符串的向量进行线性变换和函数变换，确定每个第二字符串的权重，任一第二字符串的权重用于表征该任一第二字符串在第三文本中的重要程度；以及从第三文本中删除权重满足第三条件的第二字符串，得到第一文本。
可选地，特征信息包括以下至少一项：文本信息，历史使用信息，上下文信息，语言概率信息。
本申请实施例提供的电子设备，电子设备在获取包括M个第一字符的第一文本之后，可以根据每个第一字符的特征信息，确定每个第一字符的权重。然后，电子设备可以从第二文本中删除目标字符，得到目标文本。其中，任一第一字符的权重用于表征该任一第一字符在第一文本中的重要程度，该第二文本为该第一文本的全部或部分文本，该目标字符为权重满足第一条件的第一字符。通过上述方案，在电子设备对长文本(如第一文本)进行搜索的情况下，电子设备可以根据第一文本中每个第一字符的特征信息，确定每个第一字符的权重。然后，电子设备可以删除权重满足第一条件的第一字符，从而可以以词为单位精细地简化第一文本，保留第一文本中的重要字符。如此，使得电子设备在对长文本进行搜索的过程中，可以仅搜索第一文本中的重要字符，进而能够在提高搜索效率的同时提高搜索正确率。
本实施例中各种实现方式具有的有益效果具体可以参见上述方法实施例中相应实现方式所具有的有益效果，为避免重复，此处不再赘述。
应理解的是，本申请实施例中，输入单元104可以包括图形处理器(Graphics Processing Unit，GPU)1041和麦克风1042，图形处理器1041对在 视频捕获模式或图像捕获模式中由图像捕获装置(如摄像头)获得的静态图片或视频的图像数据进行处理。显示单元106可包括显示面板1061，可以采用液晶显示器、有机发光二极管等形式来配置显示面板1061。用户输入单元107包括触控面板1071以及其他输入设备1072。触控面板1071，也称为触摸屏。触控面板1071可包括触摸检测装置和触摸控制器两个部分。其他输入设备1072可以包括但不限于物理键盘、功能键(比如音量控制按键、开关按键等)、轨迹球、鼠标、操作杆，在此不再赘述。存储器109可用于存储软件程序以及各种数据，包括但不限于应用程序和操作系统。处理器110可集成应用处理器和调制解调处理器，其中，应用处理器主要处理操作系统、用户界面和应用程序等，调制解调处理器主要处理无线通信。可以理解的是，上述调制解调处理器也可以不集成到处理器110中。
本申请实施例还提供一种可读存储介质，所述可读存储介质上存储有程序或指令，该程序或指令被处理器执行时实现上述文本处理方法实施例的各个过程，且能达到相同的技术效果，为避免重复，这里不再赘述。
其中，所述处理器为上述实施例中所述的电子设备中的处理器。所述可读存储介质，包括计算机可读存储介质，如计算机只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、磁碟或者光盘等。
本申请实施例另提供了一种芯片，所述芯片包括处理器和通信接口，所述通信接口和所述处理器耦合，所述处理器用于运行程序或指令，实现上述文本处理方法实施例的各个过程，且能达到相同的技术效果，为避免重复，这里不再赘述。
应理解，本申请实施例提到的芯片还可以称为系统级芯片、系统芯片、芯片系统或片上系统芯片等。
需要说明的是，在本文中，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者装置不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者装置所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括该要素的过程、方法、物品或者装置中还存在另外的相同要素。此外，需要指出的是，本申请实施方式中的方法和装置的范围不限按示出或讨论的顺序来执行功能，还可包括根据所涉及的功能按基本同时的方式或按相反的顺序来执行功能，例如，可以按不同 于所描述的次序来执行所描述的方法，并且还可以添加、省去、或组合各种步骤。另外，参照某些示例所描述的特征可在其他示例中被组合。
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到上述实施例方法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分可以以计算机软件产品的形式体现出来，该计算机软件产品存储在一个存储介质(如ROM/RAM、磁碟、光盘)中，包括若干指令用以使得一台终端(可以是手机，计算机，服务器，或者网络设备等)执行本申请各个实施例所述的方法。
上面结合附图对本申请的实施例进行了描述，但是本申请并不局限于上述的具体实施方式，上述的具体实施方式仅仅是示意性的，而不是限制性的，本领域的普通技术人员在本申请的启示下，在不脱离本申请宗旨和权利要求所保护的范围情况下，还可做出很多形式，均属于本申请的保护之内。

Claims (15)

  1. 一种文本处理方法，所述方法包括：
    获取第一文本，所述第一文本包括M个第一字符，M为正整数；
    根据每个第一字符的特征信息，确定所述每个第一字符的权重，任一第一字符的权重用于表征所述任一第一字符在所述第一文本中的重要程度；
    从第二文本中删除目标字符，得到目标文本，所述第二文本为所述第一文本的全部或部分文本，所述目标字符为权重满足第一条件的第一字符。
  2. 根据权利要求1所述的方法，其中，所述第一文本包括至少一个第一字符串，所述至少一个第一字符串包括所述M个第一字符，一个第一字符串包括至少一个第一字符；
    在从第二文本中删除目标字符之前，所述方法还包括：
    根据所述每个第一字符的权重，确定所述每个第一字符串的权重，任一第一字符串的权重用于表征所述任一第一字符串在所述第一文本中的重要程度；
    从所述第一文本中删除目标字符串，得到所述第二文本，所述目标字符串为权重满足第二条件的第一字符串。
  3. 根据权利要求2所述的方法，其中，所述根据所述每个第一字符的权重，确定所述每个第一字符串的权重，包括：
    将每个第一字符串对应的每个第一字符的权重进行线性变换和函数变换，确定所述每个第一字符串的权重。
  4. 根据权利要求1至3任一项所述的方法，其中，所述获取第一文本之前，所述方法还包括：
    对待处理的第三文本执行分句处理，得到至少一个第二字符串；
    对所述至少一个第二字符串中的每个第二字符串执行分词处理，得到至少一个第二字符；
    获取所述至少一个第二字符中的每个第二字符的向量；
    根据所述每个第二字符串对应的每个第二字符的向量，确定所述每个第二字符串的向量；
    将所述每个第二字符串的向量进行线性变换和函数变换，确定所述每个第二字符串的权重，任一第二字符串的权重用于表征所述任一第二字符串在所述第三文本中的重要程度；
    从所述第三文本中删除权重满足第三条件的第二字符串，得到所述第一文本。
  5. 根据权利要求1所述的方法，其中，所述特征信息包括以下至少一项：文本信息，历史使用信息，上下文信息，语言概率信息。
  6. 一种文本处理装置，所述文本处理装置包括：获取模块、确定模块和执行模块；
    所述获取模块，用于获取第一文本，所述第一文本包括M个第一字符，M为正整数；
    所述确定模块，用于根据每个第一字符的特征信息，确定所述每个第一字符的权重，任一第一字符的权重用于表征所述任一第一字符在所述第一文本中的重要程度；
    所述执行模块，用于从第二文本中删除目标字符，得到目标文本，所述第二文本为所述第一文本的全部或部分文本，所述目标字符为权重满足第一条件的第一字符。
  7. 根据权利要求6所述的文本处理装置，其中，所述第一文本包括至少一个第一字符串，所述至少一个第一字符串包括所述M个第一字符，一个第一字符串包括至少一个第一字符；
    所述确定模块，还用于根据所述每个第一字符的权重，确定所述每个第一字符串的权重，任一第一字符串的权重用于表征所述任一第一字符串在所述第一文本中的重要程度；
    所述执行模块，还用于从所述第一文本中删除目标字符串，得到所述第二文本，所述目标字符串为权重满足第二条件的第一字符串。
  8. 根据权利要求7所述的文本处理装置，其中，所述确定模块，具体用于将每个第一字符串对应的每个第一字符的权重进行线性变换和函数变换，确定所述每个第一字符串的权重。
  9. 根据权利要求6至8任一项所述的文本处理装置，其中，
    所述执行模块，还用于对待处理的第三文本执行分句处理，得到至少一个第二字符串；以及对所述至少一个第二字符串中的每个第二字符串执行分词处理，得到至少一个第二字符；
    所述获取模块，还用于获取所述至少一个第二字符中的每个第二字符的向量；
    所述确定模块，还用于根据所述获取模块获取的所述每个第二字符串对应的每个第二字符的向量，确定所述每个第二字符串的向量；以及将所述每个第二字符串的向量进行线性变换和函数变换，确定所述每个第二字符串的权重， 任一第二字符串的权重用于表征所述任一第二字符串在所述第三文本中的重要程度；
    所述执行模块，还用于从所述第三文本中删除权重满足第三条件的第二字符串，得到所述第一文本。
  10. 根据权利要求6所述的文本处理装置，其中，所述特征信息包括以下至少一项：文本信息、历史使用信息、上下文信息和语言概率信息。
  11. 一种电子设备，包括处理器，存储器及存储在所述存储器上并可在所述处理器上运行的程序或指令，所述程序或指令被所述处理器执行时实现如权利要求1至5中任一项所述的文本处理方法的步骤。
  12. 一种可读存储介质，所述可读存储介质上存储程序或指令，所述程序或指令被处理器执行时实现如权利要求1至5中任一项所述的文本处理方法的步骤。
  13. 一种计算机程序产品，所述计算机程序产品被至少一个处理器执行以实现如权利要求1-5任一项所述的文本处理方法的步骤。
  14. 一种芯片，所述芯片包括处理器和通信接口，所述通信接口和所述处理器耦合，所述处理器用于运行程序或指令，实现如权利要求1-5任一项所述的文本处理方法的步骤。
  15. 一种电子设备，所述电子设备被配置成用于执行如权利要求1-5任一项所述的文本处理方法的步骤。
PCT/CN2022/095648 2021-05-31 2022-05-27 文本处理方法、装置和电子设备 Ceased WO2022253138A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110603987.6A CN113360613A (zh) 2021-05-31 2021-05-31 文本处理方法、装置和电子设备
CN202110603987.6 2021-05-31

Publications (1)

Publication Number Publication Date
WO2022253138A1 true WO2022253138A1 (zh) 2022-12-08

Family

ID=77530559

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/095648 Ceased WO2022253138A1 (zh) 2021-05-31 2022-05-27 文本处理方法、装置和电子设备

Country Status (2)

Country Link
CN (1) CN113360613A (zh)
WO (1) WO2022253138A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2025140159A1 (zh) * 2023-12-29 2025-07-03 维沃移动通信有限公司 文本处理方法、装置、电子设备及存储介质

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113360613A (zh) * 2021-05-31 2021-09-07 维沃移动通信有限公司 文本处理方法、装置和电子设备

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110232156A (zh) * 2019-05-31 2019-09-13 三角兽（北京）科技有限公司 基于长文本的信息推荐方法及装置
CN110704621A (zh) * 2019-09-25 2020-01-17 北京大米科技有限公司 文本处理方法、装置及存储介质和电子设备
US20200050638A1 (en) * 2018-08-12 2020-02-13 Parker Douglas Hancock Systems and methods for analyzing the validity or infringment of patent claims
CN111680510A (zh) * 2020-07-07 2020-09-18 腾讯科技（深圳）有限公司 文本处理方法、装置、计算机设备以及存储介质
CN112711695A (zh) * 2020-12-29 2021-04-27 网易传媒科技（北京）有限公司 基于内容的搜索建议生成方法及装置
CN112784009A (zh) * 2020-12-28 2021-05-11 北京邮电大学 一种主题词挖掘方法、装置、电子设备及存储介质
CN113360613A (zh) * 2021-05-31 2021-09-07 维沃移动通信有限公司 文本处理方法、装置和电子设备

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103886063B (zh) * 2014-03-18 2017-03-08 国家电网公司 一种文本检索方法和装置
CN111274808B (zh) * 2020-02-11 2023-07-04 支付宝(杭州)信息技术有限公司 文本检索方法、模型训练方法、文本检索装置及存储介质

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200050638A1 (en) * 2018-08-12 2020-02-13 Parker Douglas Hancock Systems and methods for analyzing the validity or infringment of patent claims
CN110232156A (zh) * 2019-05-31 2019-09-13 三角兽（北京）科技有限公司 基于长文本的信息推荐方法及装置
CN110704621A (zh) * 2019-09-25 2020-01-17 北京大米科技有限公司 文本处理方法、装置及存储介质和电子设备
CN111680510A (zh) * 2020-07-07 2020-09-18 腾讯科技（深圳）有限公司 文本处理方法、装置、计算机设备以及存储介质
CN112784009A (zh) * 2020-12-28 2021-05-11 北京邮电大学 一种主题词挖掘方法、装置、电子设备及存储介质
CN112711695A (zh) * 2020-12-29 2021-04-27 网易传媒科技（北京）有限公司 基于内容的搜索建议生成方法及装置
CN113360613A (zh) * 2021-05-31 2021-09-07 维沃移动通信有限公司 文本处理方法、装置和电子设备

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2025140159A1 (zh) * 2023-12-29 2025-07-03 维沃移动通信有限公司 文本处理方法、装置、电子设备及存储介质

Also Published As

Publication number Publication date
CN113360613A (zh) 2021-09-07

Similar Documents

Publication Publication Date Title
CN113378556B (zh) 提取文本关键字的方法及装置
US12229499B1 (en) Computer-implemented presentation of synonyms based on syntactic dependency
EP3958145A1 (en) Method and apparatus for semantic retrieval, device and storage medium
CN111931501B (zh) 一种基于人工智能的文本挖掘方法、相关装置及设备
CN107608532B (zh) 一种联想输入方法、装置及电子设备
CN111401033B (zh) 事件抽取方法、事件抽取装置和电子设备
CN112560479A (zh) 摘要抽取模型训练方法、摘要抽取方法、装置和电子设备
CN112631437B (zh) 信息推荐方法、装置及电子设备
CN114417871B (zh) 模型训练及命名实体识别方法、装置、电子设备及介质
CN113407738B (zh) 一种相似文本检索方法、装置、电子设备和存储介质
CN111339267A (zh) 基于知识图谱的问答方法及系统、计算机设备及介质
CN102567509A (zh) 视觉信息辅助的即时通信方法及系统
CN112269475A (zh) 文字显示方法、装置和电子设备
WO2022257840A1 (zh) 信息显示方法、装置、电子设备及可读存储介质
CN108287875A (zh) 人物共现关系确定方法、专家推荐方法、装置及设备
CN113822072A (zh) 关键词抽取方法、装置及电子设备
CN111858905A (zh) 模型训练方法、信息识别方法、装置、电子设备及存储介质
CN113360769A (zh) 信息查询方法、装置、电子设备以及存储介质
WO2022253138A1 (zh) 文本处理方法、装置和电子设备
CN111538830B (zh) 法条检索方法、装置、计算机设备及存储介质
CN113821669A (zh) 搜索方法、装置、电子设备和存储介质
CN112417095A (zh) 语音消息处理方法和装置
CN113392177B (zh) 一种关键词获取方法、装置、电子设备及存储介质
CN112148988B (zh) 用于生成信息的方法、装置、设备以及存储介质
CN114443935A (zh) 内容搜索方法、装置以及电子设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22815184

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22815184

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205 DATED 07/06/2024)

122 Ep: pct application non-entry in european phase

Ref document number: 22815184

Country of ref document: EP

Kind code of ref document: A1