CN110321433B

CN110321433B - Method and device for determining text category

Info

Publication number: CN110321433B
Application number: CN201910562347.8A
Authority: CN
Inventors: 张洪
Original assignee: Advanced New Technologies Co Ltd
Current assignee: Advanced New Technologies Co Ltd; Advantageous New Technologies Co Ltd
Priority date: 2019-06-26
Filing date: 2019-06-26
Publication date: 2023-04-07
Anticipated expiration: 2039-06-26
Also published as: CN110321433A

Abstract

The embodiment of the specification provides a method and a device for determining text types, wherein the method comprises the following steps: acquiring a target text set; the target text set comprises a plurality of target texts which are not subjected to category labeling and a plurality of target texts which are subjected to category labeling; determining a target text subset corresponding to each word in the target text set; wherein, a word corresponds to a target text subset, and each target text in the target text subset comprises a word; calculating a text similarity value between any two target texts in the target text subsets aiming at each target text subset; and determining the category corresponding to each target text in the target text set according to a set text classification algorithm based on each text similarity value and a plurality of target texts subjected to category labeling.

Description

Method and device for determining text type

Technical Field

The present application relates to the field of data processing technologies, and in particular, to a method and an apparatus for determining a text type.

Background

With the rapid development of information technology, more and more fields begin to use machines to perform complex tasks that usually require human intelligence to complete, i.e., more and more fields begin to apply artificial intelligence. In order to implement application of artificial intelligence in each field, training of a corresponding model is generally required, a large amount of sample data is required to be used in model training, and the sample data is also required to be labeled according to actual requirements, for example, if the sample data is a text and the trained model is a classification model, the category of the text is required to be labeled, and the like.

However, when training a model, a large number of samples need to be labeled, and therefore, a technical solution is needed to be provided to rapidly and accurately label the samples.

Disclosure of Invention

The embodiment of the present specification aims to provide a method and an apparatus for determining a text category, wherein when determining a text category, an acquired target text set is divided into a plurality of target text subsets according to words contained in each target text, and each target text contained in each target text subset has at least one common word, so that the target text contained in each target text subset is a target text having an association relationship, and the target text between different target text subsets is considered to have no association relationship, so that when calculating a text similarity value between target texts, only the text similarity value between the target texts in each target text subset needs to be calculated, and the calculation of the text similarity value does not need to be performed on the target text between different target text subsets, thereby greatly reducing the calculation workload of the text similarity value, shortening the time consumption for calculating the text similarity value, further shortening the time for determining the text category, and improving the efficiency for determining the text category; moreover, the method provided by the embodiment of the specification can be automatically realized through a machine, and compared with the method for classifying the texts through manual work, the method is high in accuracy.

In order to solve the above technical problem, the embodiments of the present specification are implemented as follows:

an embodiment of the present specification provides a method for determining a text category, including:

acquiring a target text set; the target text set comprises a plurality of target texts which are not subjected to category labeling and a plurality of target texts which are subjected to category labeling;

determining a target text subset corresponding to each word in the target text set; wherein, a word corresponds to a target text subset, and each target text in the target text subset contains the word;

calculating a text similarity value between any two target texts in the target text subsets aiming at each target text subset;

and determining the category corresponding to each target text in the target text set according to a set text classification algorithm based on each text similarity value and the plurality of target texts subjected to category labeling.

An embodiment of the present specification further provides an apparatus for determining a text category, including:

the acquisition module is used for acquiring a target text set; the target text set comprises a plurality of target texts which are not subjected to category labeling and a plurality of target texts which are subjected to category labeling;

a first determining module, configured to determine a target text subset corresponding to each word in the target text set; wherein, a word corresponds to a target text subset, and each target text in the target text subset contains the word;

the calculation module is used for calculating a text similarity value between any two target texts in the target text subsets aiming at each target text subset;

and the second determining module is used for determining the category corresponding to each target text in the target text set according to a set text classification algorithm based on each text similarity value and the plurality of target texts subjected to category marking.

a processor; and

a memory arranged to store computer executable instructions that, when executed, cause the processor to:

Embodiments of the present specification also provide a storage medium for storing computer-executable instructions, which when executed implement the following processes:

According to the technical scheme in the embodiment, when the text type is determined, the obtained target text set is divided into a plurality of target text subsets according to words contained in each target text, and the target text contained in each target text subset has at least one common word, so that the target text contained in each target text subset is a target text with an association relationship, and the target text between different target text subsets is considered to have no association relationship, therefore, when the text similarity value between the target texts is calculated, only the text similarity value between the target texts in each target text subset needs to be calculated, and the text similarity value does not need to be calculated for the target text between different target text subsets, so that the calculation workload of the text similarity value is greatly reduced, the time consumption for calculating the text similarity value is shortened, the time consumption for determining the text type is further shortened, and the efficiency for determining the text type is improved; moreover, the method provided by the embodiment of the specification can be automatically realized through a machine, and compared with the method for classifying texts through manual work, the method is high in accuracy.

Drawings

In order to more clearly illustrate the embodiments of the present specification or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the description below are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art without creative efforts.

FIG. 1 is a flowchart of a method for determining a text category according to an embodiment of the present disclosure;

fig. 2 is a schematic diagram of a similar target text corresponding to a target text 1 in the method for determining a text category provided in the embodiment of the present specification;

FIG. 3 is a second flowchart of a method for determining a text category according to an embodiment of the present disclosure;

FIG. 4 is a third flowchart of a method for determining a text category according to an embodiment of the present disclosure;

FIG. 5 is a flowchart of a method for determining text type according to an embodiment of the present disclosure;

FIG. 6 is a fifth flowchart of a method for determining text categories according to an embodiment of the present disclosure;

FIG. 7 is a schematic block diagram illustrating an apparatus for determining a text category according to an embodiment of the present disclosure;

fig. 8 is a schematic structural diagram of an apparatus for determining a text category according to an embodiment of the present specification.

Detailed Description

In order to make those skilled in the art better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making creative efforts shall fall within the protection scope of the present application.

The idea of the embodiment of the present specification is that, according to words contained in each target text in a target text set, target texts containing at least one same word are divided into the same target text subset, and when text similarity values between target texts are calculated, only the text similarity values between the target texts in each target text subset need to be calculated, so that calculation of the text similarity values between target texts which are not related is avoided, workload for calculating the text similarity values is reduced, time consumed for determining text categories is shortened, and efficiency for determining text categories is improved. Based on this, embodiments of the present specification provide a method, an apparatus, a device, and a storage medium for determining a text type, which are described in detail below.

The execution subject of the method provided by the embodiment of the specification is a device for determining the text type, and the device can be a device installed on intelligent equipment such as a computer, a computer and the like, and can also be a device installed on a server.

Fig. 1 is a flowchart of a method for determining a text category according to an embodiment of the present disclosure, where the method shown in fig. 1 at least includes the following steps:

102, acquiring a target text set; the target text set comprises a plurality of target texts which are not subjected to category labeling and a plurality of target texts which are subjected to category labeling.

In the embodiments of the present specification, each target text may be obtained from a database or a server capable of generating the target text, so as to form a target text set. The database may be an offline database or an online database. The target text can be a sentence, a paragraph or an article. For example, in one embodiment, if a model for classifying a commodity needs to be trained, the target text may be description information of the commodity. The specific content of the target text in the target text set is related to an actual application scenario, and the specific content of the target text is not illustrated in this specification one by one.

Step 104, determining a target text subset corresponding to each word in the target text set; each target text in the target text subset comprises the word.

The words in the target text set may be understood as all the words contained in all the target texts in the target text set. In a specific implementation, word segmentation processing may be performed on each target text in the target text set to obtain words included in each target text, and the words included in each obtained target text are determined as the words included in the target text set. In specific implementation, word segmentation processing can be performed on each target text by adopting word segmentation algorithms such as AliWS and jieba. For example, a target text is "dragon-brand 9.5 cm double-defense plasterboard", and words obtained by performing word segmentation on the target text are "dragon-brand", "9.5 cm", "double-defense" and "plasterboard".

For ease of understanding, the following description will be made by way of example.

For example, in one specific embodiment, the target text set includes five target texts, namely, a target text 1, a target text 2, a target text 3, a target text 4, and a target text 5, and the five target texts are subjected to word segmentation processing to obtain words contained in each target text, for example, words obtained by performing word segmentation on the target text 1 are a word 1, a word 2, a word 3, a word 5, a word 7, and a word 9, words obtained by performing word segmentation on the target text 2 are a word 2, a word 4, a word 5, a word 8, a word 9, and a word 10, words obtained by performing word segmentation on the target text 3 are a word 1, a word 2, a word 3, a word 5, a word 6, a word 8, and a word 10, words obtained by performing word segmentation on the target text 4 are a word 3, a word 4, a word 6, a word 7, words 8, and 9, and words obtained by performing word segmentation on the target text 5 are a word 1, a word 2, a word 4, a word 5, a word 7, a word 9, and a word 10;

all the words included in the above five target texts are word 1, word 2, word 3, word 4, word 5, word 6, word 7, word 8, word 9, and word 10, and therefore, the words corresponding to the target text set can be determined as word 1, word 2, word 3, word 4, word 5, word 6, word 7, word 8, word 9, and word 10.

In this embodiment of the present specification, after obtaining a word corresponding to a target text set, for each word corresponding to the target text set, a target text including the word is screened from all target texts included in the target text set, and the screened target texts are combined into a target text subset corresponding to the word. In this way, the target text subset corresponding to each word in the target text set can be determined, that is, the target text set can be divided into a plurality of target text subsets, and the target text in each target text subset includes one same word.

For example, continuing with the above example, for word 1, the target text in the target text set that includes word 1 has target text 1 and target text 3, and therefore the target text subset corresponding to word 1 is { target text 1, target text 3}; the target text containing the word 2 in the target text set comprises a target text 1, a target text 2, a target text 3 and a target text 5, so that the target text subset corresponding to the word 2 is { the target text 1, the target text 2, the target text 3 and the target text 5}; through the above manner, it can be determined that the target text subset corresponding to the word 3 is { target text 1, target text 3, target text 4}, the target text subset corresponding to the word 4 is { target text 2, target text 4, target text 5}, the target text subset corresponding to the word 5 is { target text 1, target text 2, target text 3, target text 5}, the target text subset corresponding to the word 6 is { target text 3, target text 4}, the target text subset corresponding to the word 7 is { target text 1, target text 4, target text 5}, the target text subset corresponding to the word 8 is { target text 2, target text 3, target text 4}, the target text subset corresponding to the word 9 is { target text 1, target text 2, target text 4, target text 5}, and the target text subset corresponding to the word 10 is { target text 2, target text 3, target text 5}.

In addition, it should be noted that, in the embodiment of the present specification, the number of the obtained target text subsets is equal to the number of words included in the target text set, for example, if the target text set includes M (where M is a positive integer) words, the number of the determined target text subsets is also M, and the target texts belonging to the same target text subset all include a common word, and of course, for a certain target text, it may belong to multiple target text subsets at the same time.

And 106, calculating a text similarity value between any two target texts in the target text subsets aiming at each target text subset.

In specific implementation, a distributed framework can be adopted to calculate the similarity values between the target texts in each target text subset in parallel, for example, map Reduce and the like can be adopted, so that the calculation time can be greatly shortened.

In the embodiment of the present specification, the target text set is divided into a plurality of target text subsets, and each target text belonging to the same target text subset has the same word, so that each target text belonging to the same target text subset can be considered to have a certain degree of association, and target texts between different target text subsets are considered to have no degree of association. Therefore, when the text similarity value between the target texts is performed, only the text similarity value between any two target texts in each target text subset needs to be calculated, and the text similarity value does not need to be calculated for the target texts in different target text subsets. Therefore, the calculation amount of the text similarity value can be greatly reduced, the time consumption is shortened, and the working efficiency is improved.

In addition, in specific implementation, the target text can be divided into a plurality of target text subsets, and when the text similarity value is calculated, the similarity value between any two target texts in each target text subset is independently calculated, so that each target text subset can be processed in parallel, time consumption for calculating the text similarity value can be further shortened, and the working efficiency is improved.

And 108, determining the category corresponding to each target text in the target text set according to a set text classification algorithm based on each text similarity value and the plurality of target texts subjected to category labeling.

In specific implementation, because the target text set includes the target text with the category already labeled, when determining the category corresponding to each target text in the target text set, the category between each target text can be determined based on the text similarity value between each target text and the target text with the category labeled. For example, the category of the target text corresponding to the maximum text similarity value may be determined as the category corresponding to a certain target text, or the categories to which most of the target texts in the target texts having a similarity relationship with the certain target text belong may be determined as the categories corresponding to the target text.

Of course, in a specific embodiment, the set text classification algorithm may be a k-Nearest Neighbor (KNN) classification algorithm, that is, in step 108, based on each text similarity value and the target text with class label, a KNN classification algorithm is used to determine the class corresponding to each target text in the target text set.

The method provided by the embodiment of the specification realizes automatic labeling of the text category, and has higher accuracy compared with the method for labeling the text category in a manual mode; when the similarity value between the target texts is calculated, the target text set is divided into a plurality of target text subsets according to the words contained in the target texts in the target text set, and only the text similarity value between the target texts in each target text subset needs to be calculated, so that the text similarity value between target texts which are not related is avoided being calculated, the workload of calculating the text similarity value is reduced, the time consumption for determining the text category is shortened, and the efficiency for determining the text category is improved.

In step 106, a text similarity value between any two target texts may be calculated based on an existing similarity algorithm. For ease of understanding, the above-described calculation process will be described by taking the example of calculating the text similarity value between the target text 1 and the target text 2. One possible way of calculation is as follows:

first, a text similarity value between the target text 1 and the target text 2 is calculated based on the longest common word string by the following formula 1:

/>

wherein, in the above formula 1, sim _lcs (S1, S2) represents a text similarity value between the target text 1 and the target text 2 calculated based on the longest common word string, S1 represents the target text 1, S2 represents the target text 2, len _lcs Indicates the length, min, of the longest common string _len Indicating the length of one target text having a smaller character length between target text 1 and target text 2, i.e., min _len = min { len (s 1), len (s 2) }, where len (s 1) denotes the character length of target text 1 and len (s 2) denotes the character length of target text 2.

Secondly, calculating a text similarity value between the target text 1 and the target text 2 according to the edit distance between the target text 1 and the target text 2, as shown in the following formula 2:

wherein len _ed Indicating the edit distance, min, of both target text 1 and target text 2 _len A length of one target text indicating that the character length between the target text 1 and the target text 2 is small, i.e., min _len Min { len (s 1), len (s 2) }, where len (s 1) denotes the character length of the target text 1 and len (s 2) denotes the character length of the target text 2.

Again, the text similarity value between the target text 1 and the target text 2 is calculated by the following formula 3 based on the Jaccard similarity coefficient:

where | s1 ≧ s2| represents the number of word intersections between both the target text 1 and the target text 2, and | s1 ≦ s2| represents the number of word unions between both the target text 1 and the target text 2.

Finally, performing fusion processing, namely weighted summation, on the text similarity values calculated by the three methods, wherein the value obtained by weighted calculation is used as the text similarity value between the target text 1 and the target text 2, and the formula of the fusion processing is shown as the following formula 4:

sim(s1,s2)＝A*sim _lcs (s1,s2)+B*sim _ed (s1,s2)+C*sim _jaccard (s 1, s 2) formula 4

In the above formula 4, A, B and C both represent weight coefficients, and the sum of a + B + C is 1. In a possible implementation manner, the value of a is 0.3, the value of b is 0.3, and the value of c is 0.4, and of course, the specific value of A, B, C may be set according to actual requirements, which is only an exemplary description here and does not constitute a limitation on the value of A, B, C.

In the embodiments of the present specification, the three methods for calculating similarity values are only used as an example for illustration, and besides, other methods for calculating similarity values may be used to calculate a plurality of similarity values respectively and perform fusion processing, and of course, three similarity values may be used to perform fusion processing, or another number of similarity values may be used to perform fusion processing, which is not limited in the embodiments of the present specification.

In the method provided by the embodiment of the specification, a plurality of different similarity values are calculated by adopting a plurality of different similarity calculation modes, then the similarity values are fused to determine the text similarity value between the target texts, and compared with the method adopting a single similarity algorithm for calculation, the calculated text similarity value has better overall balance and higher accuracy.

Specifically, in the step 108, determining the category corresponding to each target text in the target text set according to a set text classification algorithm based on each text similarity value and a plurality of target texts subjected to category labeling, at least includes the following steps:

determining a category corresponding to each target text in a plurality of target texts which are not subjected to category marking according to a set text classification algorithm based on each text similarity value and a plurality of target texts which are subjected to category marking;

and (c) a second step of,

and updating the category corresponding to each target text in the plurality of target texts subjected to category labeling according to a set text classification algorithm based on the text similarity values and the plurality of target texts subjected to category labeling.

That is, in the embodiment of the present specification, the category corresponding to the target text which is not subjected to category labeling in the target text set may be determined, the category of the target text which is subjected to category labeling may also be updated, and by updating the category of the target text which is subjected to category labeling, some target texts which are incorrectly labeled or inaccurate in the target text which is subjected to category labeling may be found in time and changed in time, so that the accuracy of subsequent model training is further improved.

In specific implementation, in the step 108, based on each text similarity value and a plurality of target texts which have been subjected to category labeling, according to a set text classification algorithm, determining a category corresponding to each target text in a target text set, including the following steps one and two;

step one, aiming at each target text in a target text set, determining at least one similar target text corresponding to the target text based on each text similarity value;

and secondly, determining the category corresponding to the target text or updating the category corresponding to the target text by using a KNN classification algorithm according to the similar target text which is subjected to category marking in the similar target text corresponding to the target text.

The KNN classification algorithm selects K nearest neighbors corresponding to the target text (i.e., K similar target texts with the highest text similarity value to the target text), and determines the category corresponding to the target text based on the proportion of each category in the similar target texts with category labeling in the middle of the K nearest neighbors.

For example, in a specific embodiment, the similar target texts corresponding to the target text 1 include a target text 2, a target text 3, a target text 4, a target text 5, a target text 6, and a target text 7, where the target text 2, the target text 3, the target text 6, and the target text 7 have been subjected to category labeling, and the target text 2, the target text 3, and the target text 7 all belong to category 1, the target text 6 belongs to category 2, a text similarity value between the target text 1 and the target text 2 is 90%, a text similarity value between the target text 1 and the target text 3 is 87%, a text similarity value between the target text 1 and the target text 4 is 80%, a text similarity value between the target text 1 and the target text 5 is 79%, a text similarity value between the target text 1 and the target text 6 is 82%, and a text similarity value between the target text 1 and the target text 7 is 95%, and based on this, a schematic diagram of the similar target texts corresponding to the target text 1 is shown in fig. 2. If the value of K is 4, it may be determined that the K nearest neighbor similar target text corresponding to the target text 1 is the target text 2, the target text 3, the target text 6, and the target text 7, that is, the target text located in the box in fig. 2, and when the category corresponding to the target text 1 is determined, it is determined based on the categories corresponding to the target text 2, the target text 3, the target text 6, and the target text 7, because the target text 2, the target text 3, and the target text 7 all belong to category 1, and the target text 6 belongs to category 2, that is, in the K nearest neighbor similar target text of the target text 1, the nearest neighbor similar target text of 3/4 belongs to category 1,1/4 is nearest neighbor to category 2, it may be determined that the category described in the target text 1 is category 1.

If the target text 1 is the target text subjected to category labeling and the category to which the target text 1 belongs is category 3, since the nearest neighbor similar target texts of 3/4 of the K nearest neighbor similar target texts of the target text 1 all belong to category 1, it can be considered that the category to which the target text 1 originally labeled belongs is inaccurate, and the category to which the target text 1 belongs is updated to category 1.

In addition, it should be noted that, in the embodiment of the present specification, the value obtained by using the KNN classification algorithm K is dynamically adjusted, that is, the value can be determined according to the similar target texts that have been subjected to category labeling in the similar target texts corresponding to each target text. For example, in a specific embodiment, the minimum value of K may be set to be 3, if 3 nearest neighboring similar target texts corresponding to a certain target text are all target texts without class labeling when determining a class to which the target text belongs, in this case, when determining a class corresponding to the target text, the value of K may be adjusted to be 4, and if 4 nearest neighboring similar target texts corresponding to the target text are all target texts without class labeling, the value of K is continuously increased.

Fig. 3 is a second flowchart of a method for determining a text category according to an embodiment of the present disclosure, where the method shown in fig. 3 at least includes the following steps:

step 302, acquiring a target text set; the target text set comprises a plurality of target texts which are not subjected to category labeling and a plurality of target texts which are subjected to category labeling.

Step 304, determining a target text subset corresponding to each word in the target text set; each target text in the target text subset contains the word.

Step 306, for each target text subset, calculating a text similarity value between any two target texts in the target text subset.

And 308, aiming at each target text in the target text set, determining at least one similar target text corresponding to the target text based on each text similarity value.

And 310, determining the category corresponding to the target text or updating the category corresponding to the target text currently by using a KNN classification algorithm according to the similar target texts which are subjected to category labeling in the similar target texts corresponding to the target text.

In a specific implementation, in the step one, determining a similar target text corresponding to each target text in the target text set may be implemented through the following processes:

aiming at each target text in the target text set, screening text similarity values meeting set conditions from the text similarity values of the target text and other target texts; the set condition comprises that the text similarity value is larger than or equal to a set threshold value, or is ranked in the top N according to numerical values from large to small; wherein N is a positive integer; and determining a similar target text corresponding to the target text based on the text similarity value obtained after screening.

In the embodiment of the present specification, the text similarity value satisfying the set condition may be screened at least in the following three ways:

firstly, aiming at all text similarity values corresponding to a certain target text, comparing each text similarity value with the set threshold value respectively, and screening out the text similarity values which are greater than or equal to the set threshold value;

secondly, sequencing all the text similarity values according to a descending order aiming at all the text similarity values corresponding to a certain target text, and intercepting the first N text similarity values according to a front-to-back order after sequencing;

thirdly, all the text similarity values corresponding to a certain target text are sequenced from small to large, and after the sequencing is completed, N text similarity values are intercepted according to the sequence from back to front.

And after the text similarity values meeting the set conditions are screened out, the target text corresponding to the text similarity values is used as the similar target text of the target text.

For the sake of understanding, the following description will be given by way of example.

For example, in one embodiment, a certain target text subset includes target text 1, target text 2, target text 3, target text 4, and target text 5, the text similarity value between target text 1 and target text 2 is 93%, the text similarity value between target text 1 and target text 3 is 89%, the text similarity value between target text 1 and target text 4 is 63%, the text similarity value between target text 1 and target text 5 is 94%, if the set threshold is 75%, the text similarity values greater than or equal to the set threshold are 93%, 89%, and 94%, that is, the screened text similarity values satisfying the set condition are 93%, 89%, and 94%,93% is the text similarity value between target text 1 and target text 2, 89% is the text similarity value between target text 1 and target text 3, and 94% is the text similarity value between target text 1 and target text 5. Therefore, the similar target texts of the determined target text 1 are the target text 2, the target text 3 and the target text 5.

In addition, in the embodiments of the present specification, specific values of the setting threshold and N may be set according to an actual application scenario, and the embodiments of the present specification do not limit the specific values of the setting threshold and N.

In the embodiment of the present specification, by performing the process of screening the text similarity values, some target texts with very low text similarity values can be screened out, so that the similar target texts corresponding to each target text can be cut, interference of the target texts with very low similarity values on the whole is reduced, and the work efficiency is improved.

Fig. 4 is a third flowchart of a method for determining a text category according to an embodiment of the present disclosure, where the method shown in fig. 4 at least includes the following steps:

step 402, acquiring a target text set; the target text set comprises a plurality of target texts which are not subjected to category labeling and a plurality of target texts which are subjected to category labeling.

Step 404, determining a target text subset corresponding to each word in the target text set; each target text in the target text subset contains the word.

Step 406, calculating a text similarity value between any two target texts in each target text subset.

And step 408, for each target text in the target text set, screening the text similarity values of the target text and other target texts, wherein the text similarity values are greater than or equal to a set threshold value.

And step 410, determining a similar target text corresponding to the target text based on the screened text similarity value.

Step 412, according to the similar target texts with the category labels in the similar target texts corresponding to the target text, determining the category corresponding to the target text or updating the category corresponding to the target text currently by using a KNN classification algorithm.

In the embodiment of the present specification, the purpose of dividing the target text set into a plurality of target text subsets is to reduce the workload and improve the efficiency. However, when determining each target text subset, since the target text may contain some words having no meaning, such as "what", "yes", "not", "cm", "number", which are not related to the current application scenario, and if the similarity value between the target texts in the target text subsets corresponding to these words is calculated, then the target text subsets corresponding to such words may be deleted before performing step 106. Moreover, in the embodiment of the present specification, for some target text subsets including a larger number of target texts, in order to further improve efficiency, the target text subsets including a larger number of target texts may be split into two or more target text subsets, and then the calculation of the text similarity value is performed.

Therefore, in this embodiment, before performing step 106, the method provided in this embodiment further includes:

deleting the target text subset corresponding to the words which are irrelevant to the scene;

and/or the presence of a gas in the gas,

splitting a target text subset with the number of target texts being larger than or equal to a set number into at least two target text subsets; and the target texts in each split target text subset comprise at least one other common word except the word corresponding to the target text subset before splitting.

Of course, in specific implementation, before the step 106 is executed, only the operation of deleting the target text subset corresponding to the words that are not related to the scene may be executed; the operation of splitting the target text subset containing the target text with the number larger than or equal to the set number into at least two target text subsets can also be executed; or, the operation of deleting the target text subsets corresponding to the words irrelevant to the scene and the operation of splitting the target text subsets containing the target texts with the number larger than or equal to the set number into at least two target text subsets are executed at the same time.

It should be noted that the above-mentioned words not related to the scene can be understood as stop words without any meaning, such as "what", "this", etc.

Specifically, when splitting a target text subset containing a large number of target texts, a bigram policy, a trigram policy, or the like may be used to control the size of each split target text subset.

To facilitate understanding of the above-described splitting process of the target text subset, the following description will be given by way of example.

For example, in one embodiment, the target text subset 1 comprises target text 1, target text 2, target text 3, target text 4 …, target text 29 and target text 30, and it is assumed that the target text subset 1 is a target text subset corresponding to "coated paper", wherein the target text 1, target text 2, …, target text 14, target text 15 and target text 16 in the target text subset further comprise the common word "matt", and the target text 17, target text 18, …, target text 29 and target text 30 further comprise the common word "morning song", so that the target text subset 1 can be split into two target text subsets, respectively denoted as target text subset 11 and target text subset 12, and the target text subset 11 comprises target text 1, target text 2, …, target text 14, target text 15 and target text 16, and target text subset 3712, and target text subset 11 comprises target text subset 18, target text subset 3732, target text subset 32 and target text subset 3732.

Fig. 5 is a fourth flowchart of a method for determining a text category according to an embodiment of the present disclosure, where the method shown in fig. 5 at least includes the following steps:

step 502, acquiring a target text set; the target text set comprises a plurality of target texts which are not subjected to category labeling and a plurality of target texts which are subjected to category labeling.

Step 504, determining a target text subset corresponding to each word in the target text set; one word corresponds to one target text subset, and each target text in the target text subset contains the word.

Step 506, deleting the target text subset corresponding to the words irrelevant to the scene in the target text, and splitting the target text subset with the number of the target texts greater than or equal to the set number into at least two target text subsets.

And the split target texts in each target text subset comprise at least one other common word except the word corresponding to the target text subset before splitting.

Step 508, calculating a text similarity value between any two target texts in each target text subset.

Step 510, for each target text in the target text set, screening a text similarity value greater than or equal to a set threshold value from the text similarity values of the target text and other target texts.

And step 512, determining a similar target text corresponding to the target text based on the screened text similarity value.

And 514, determining the category corresponding to the target text or updating the category corresponding to the target text currently by using a KNN classification algorithm according to the similar target text which is subjected to category labeling in the similar target text corresponding to the target text.

In the embodiment of the present specification, before the text similarity value is calculated, by deleting a meaningless target text subset or splitting a target text subset containing a large number of target texts, the workload of calculating the text similarity value can be further reduced, thereby improving the work efficiency.

Generally, in specific implementation, after the target text set is obtained, some meaningless words or some words with different formats may exist in each target text in the target text set, so that in order to improve the quality of the target text, some unnecessary workload in the execution process of subsequent steps is reduced. In this embodiment of the present specification, before performing step 104, the following steps are further included:

preprocessing each target text in the target text set;

wherein the pretreatment comprises at least one of the following treatments:

deleting stop words and nonsense words in the target text, deleting additional description information in the target text, deleting special characters in the target text, and unifying character codes in the target text.

Of course, the above pre-treatment may also include other treatments, and the embodiments in this specification are not listed.

The nonsense words can be understood as words that do not affect the classification, such as dates. For example, in one embodiment, the target text is "2016, 12, month, and 1 day prefabricated plate", and the date "2016, 12, month, and 1 day" in the target text is a nonsense word, so that the date in the target text can be deleted.

The above-mentioned additional description information may be understood as some description information in parentheses in the target text, for example, the target text is "laundry detergent (exclusive to sock)", and the "exclusive to sock" in parentheses is an additional description of "laundry detergent", and therefore, the additional description information of "laundry detergent" in parentheses may be deleted.

The special characters mentioned above can be understood as punctuation marks, illegal characters, etc., e.g. the target text contains "! Gold ", in the target text"! The characters are special characters and need to be deleted when the target text is preprocessed.

The above-mentioned character codes in the same target text can be understood as "kilograms" if the target text includes the specification, unit, model, weight, etc. of some commodities, for example, "kilograms", "KG", and "KG" are all unified as "kilograms".

Fig. 6 is a fifth flowchart of a method for determining a text category according to an embodiment of the present disclosure, where the method shown in fig. 6 at least includes the following steps:

step 602, acquiring a target text set; the target text set comprises a plurality of target texts which are not subjected to category labeling and a plurality of target texts which are subjected to category labeling.

And step 604, preprocessing each target text in the target text set.

Step 606, determining a target text subset corresponding to each word in the preprocessed target text set; each target text in the target text subset contains the word.

Step 608, deleting the target text subset corresponding to the words irrelevant to the scene in the target text, and splitting the target text subset containing the target texts with the number larger than or equal to the set number into at least two target text subsets.

And the target texts in each split target text subset comprise at least one other common word except the word corresponding to the target text subset before splitting.

Step 610, for each target text subset, calculating a text similarity value between any two target texts in the target text subset.

Step 612, for each target text in the target text set, screening the text similarity values greater than or equal to a set threshold value from the text similarity values of the target text and other target texts.

And 614, determining a similar target text corresponding to the target text based on the screened text similarity value.

Step 616, according to the similar target texts with the class labels in the similar target texts corresponding to the target text, determining the class corresponding to the target text or updating the class corresponding to the target text currently by using a KNN classification algorithm.

In the method for determining the target text category provided in the embodiment of the present specification, when determining the text category, the obtained target text set is divided into a plurality of target text subsets according to words contained in each target text, and the target text contained in each target text subset has at least one common word, so that the target text contained in each target text subset is a target text having an association relationship, and the target text between different target text subsets is considered to have no association relationship, so that when calculating the text similarity value between target texts, only the text similarity value between the target texts in each target text subset needs to be calculated, and the text similarity value does not need to be calculated for the target text between different target text subsets, thereby greatly reducing the calculation workload of the text similarity value, shortening the time consumption for calculating the text similarity value, further shortening the time consumption for determining the text category, and improving the efficiency for determining the text category; moreover, the method provided by the embodiment of the specification can be automatically realized through a machine, and compared with the method for classifying texts through manual work, the method is high in accuracy.

Corresponding to the method provided by the embodiment of the present specification, based on the same idea, an apparatus for determining a text category is further provided in the embodiment of the present specification, and is configured to execute the method provided by the embodiment of the present specification, fig. 7 is a schematic diagram illustrating module components of the apparatus for determining a text category provided by the embodiment of the present specification, where the apparatus shown in fig. 7 includes:

an obtaining module 702, configured to obtain a target text set; the target text set comprises a plurality of target texts which are not subjected to category marking and a plurality of target texts which are subjected to category marking;

a first determining module 704, configured to determine a target text subset corresponding to each word in the target text set; wherein, a word corresponds to a target text subset, and each target text in the target text subset comprises a word;

a calculating module 706, configured to calculate, for each target text subset, a text similarity value between any two target texts in the target text subset;

the second determining module 708 is configured to determine, based on each text similarity value and the plurality of target texts with category labels, a category corresponding to each target text in the target text set according to a set text classification algorithm.

Optionally, the second determining module 708 includes:

the first determining unit is used for determining a category corresponding to each target text in a plurality of target texts which are not subjected to category marking according to a set text classification algorithm based on each text similarity value and a plurality of target texts which are subjected to category marking;

and (c) a second step of,

and the updating unit is used for updating the category corresponding to each target text in the plurality of target texts subjected to category marking according to a set text classification algorithm based on each text similarity value and the plurality of target texts subjected to category marking.

Optionally, the second determining module 708 includes:

the second determining unit is used for determining at least one similar target text corresponding to the target text based on each text similarity value aiming at each target text in the target text set;

and the execution unit is used for determining the category corresponding to the target text or updating the category corresponding to the target text at present by using a k-nearest neighbor KNN classification algorithm according to the similar target text which is subjected to category marking in the similar target text corresponding to the target text.

Optionally, the second determining unit includes:

the screening subunit is used for screening the text similarity values meeting the set conditions from the text similarity values of the target text and other target texts aiming at each target text in the target text set; the set conditions comprise that the text similarity value is larger than or equal to a set threshold value, or the text similarity value is ranked in the top N according to the numerical value from large to small; wherein N is a positive integer;

and the determining subunit is used for determining a similar target text corresponding to the target text based on the text similarity value obtained after screening.

Optionally, the apparatus provided in this specification further includes:

the deleting module is used for deleting the target text subset corresponding to the words which are irrelevant to the scene;

and/or the presence of a gas in the gas,

the splitting module is used for splitting the target text subsets of which the number of the contained target texts is greater than or equal to the set number into at least two target text subsets; and the target texts in each split target text subset comprise at least one other common word except the word corresponding to the target text subset before splitting.

Optionally, the apparatus provided in this specification further includes:

the processing module is used for preprocessing each target text in the target text set;

wherein the pre-processing comprises at least one of the following processes:

The apparatus for determining a text category according to this embodiment of the present specification may further perform the method performed by the apparatus for determining a text category in fig. 1 to fig. 6, and implement the functions of the apparatus for determining a text category according to the embodiments shown in fig. 1 to fig. 6, which are not described herein again.

In the device for determining the text type provided in the embodiment of the present specification, when determining the text type, the obtained target text set is divided into a plurality of target text subsets according to words contained in each target text, and the target text contained in each target text subset has at least one common word, so that the target text contained in each target text subset is a target text having an association relationship, and the target text between different target text subsets is considered to have no association relationship, so that when calculating the text similarity value between target texts, only the text similarity value between the target texts in each target text subset needs to be calculated, and the text similarity value does not need to be calculated for the target text between different target text subsets, thereby greatly reducing the calculation workload of the text similarity value, shortening the time consumption for calculating the text similarity value, further shortening the time consumption for determining the text type, and improving the efficiency for determining the text type; moreover, the method provided by the embodiment of the specification can be automatically realized through a machine, and compared with the method for classifying texts through manual work, the method is high in accuracy.

Further, based on the methods shown in fig. 1 to fig. 6, an embodiment of the present specification further provides an apparatus for determining a text category, as shown in fig. 8.

The devices for determining the text category may vary significantly depending on configuration or performance, and may include one or more processors 801 and memory 802, where one or more stored applications or data may be stored in memory 802. Wherein the memory 802 may be a transient storage or a persistent storage. The application program stored in memory 802 may include one or more modules (not shown), each of which may include a series of computer-executable instruction messages for a device that determines a category of text. Still further, the processor 801 may be configured to communicate with the memory 802 to execute a series of computer-executable instruction information in the memory 802 on a device that determines a text category. The apparatus to determine the text category may also include one or more power supplies 803, one or more wired or wireless network interfaces 804, one or more input-output interfaces 805, one or more keyboards 806, and the like.

In one particular embodiment, an apparatus for determining a category of text includes a memory, and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a sequence of computer-executable instruction information for the apparatus for determining a category of text, and the one or more programs configured for execution by one or more processors include computer-executable instruction information for:

acquiring a target text set; the target text set comprises a plurality of target texts which are not subjected to category marking and a plurality of target texts which are subjected to category marking;

determining a target text subset corresponding to each word in the target text set; wherein, a word corresponds to a target text subset, and each target text in the target text subset comprises a word;

and determining the category corresponding to each target text in the target text set according to a set text classification algorithm based on each text similarity value and a plurality of target texts subjected to category labeling.

Optionally, when executed, the computer-executable instruction information determines, based on each text similarity value and a plurality of target texts with category labels, a category corresponding to each target text in the target text set according to a set text classification algorithm, including:

and the number of the first and second groups,

aiming at each target text in the target text set, determining at least one similar target text corresponding to the target text based on each text similarity value;

and determining the category corresponding to the target text or updating the category corresponding to the target text at present by using a k-nearest neighbor KNN classification algorithm according to the similar target text which is subjected to category marking in the similar target text corresponding to the target text.

Optionally, when executed, the computer-executable instruction information determines, for each target text in the target text set, at least one similar target text corresponding to the target text based on the text similarity value, where the determining includes:

screening text similarity values meeting set conditions from the text similarity values of the target texts and other target texts aiming at each target text in the target text set; the set conditions comprise that the text similarity value is larger than or equal to a set threshold value, or the text similarity value is ranked in the top N according to the numerical value from large to small; wherein N is a positive integer;

and determining a similar target text corresponding to the target text based on the text similarity value obtained after screening.

Optionally, before the computer-executable instruction information is executed to calculate, for each target text subset, a text similarity value between any two target texts in the target text subset, the method further includes:

and/or the presence of a gas in the gas,

Optionally, before the computer-executable instruction information determines the target text subset corresponding to each word in the target text set when executed, the following steps may be further performed:

preprocessing each target text in the target text set;

wherein the pre-processing comprises at least one of the following processes:

Further, based on the methods shown in fig. 1 to fig. 6, in a specific embodiment, the storage medium may be a usb disk, an optical disk, a hard disk, or the like, and when executed by a processor, the storage medium stores computer-executable instruction information that implements the following processes:

Optionally, when executed by the processor, the computer-executable instruction information stored in the storage medium determines, according to a set text classification algorithm, a category corresponding to each target text in the target text set based on each text similarity value and a plurality of target texts that have been subjected to category labeling, including:

and the number of the first and second groups,

and updating the category corresponding to each target text in the target texts subjected to category labeling according to a set text classification algorithm based on the text similarity values and the target texts subjected to category labeling.

Optionally, when executed by the processor, the computer-executable instruction information stored in the storage medium determines, according to a set text classification algorithm, a category corresponding to each target text in the target text set based on each text similarity value and a plurality of target texts that have undergone category labeling, where the determining includes:

Optionally, when executed by the processor, the computer-executable instruction information stored in the storage medium determines, for each target text in the target text set, at least one similar target text corresponding to the target text based on the respective text similarity value, and includes:

screening a text similarity value meeting a set condition from the text similarity values of the target text and other target texts aiming at each target text in the target text set; the set conditions comprise that the text similarity value is larger than or equal to a set threshold value, or the text similarity value is ranked in the top N according to the numerical value from large to small; wherein N is a positive integer;

Optionally, before the computer-executable instruction information stored in the storage medium is executed by the processor and the text similarity value between any two target texts in the target text subset is calculated for each target text subset, the method further includes:

and/or the presence of a gas in the atmosphere,

splitting the target text subsets of which the number of the contained target texts is greater than or equal to the set number into at least two target text subsets; and the split target texts in each target text subset comprise at least one other common word except the word corresponding to the target text subset before splitting.

Optionally, before the storage medium stores computer-executable instruction information, and when the processor determines the target text subset corresponding to each word in the target text set, the following steps may be further performed:

preprocessing each target text in the target text set;

wherein the pre-processing comprises at least one of the following processes:

When the computer-executable instruction information stored in the storage medium provided in the embodiment of the present specification is executed by the processor, when determining the text type, the obtained target text set is divided into a plurality of target text subsets according to the words contained in each target text, and the target text contained in each target text subset has at least one common word, so that the target text contained in each target text subset is a target text having an association relationship, and the target text between different target text subsets is considered to have no association relationship, so that when calculating the text similarity value between target texts, only the text similarity value between the target texts in each target text subset needs to be calculated, and the calculation of the text similarity value for the target text between different target text subsets does not need to be performed, so that the calculation workload of the text similarity value is greatly reduced, the time consumption for calculating the text similarity value is reduced, the time for determining the text type is reduced, and the efficiency for determining the text type is improved; moreover, the method provided by the embodiment of the specification can be automatically realized through a machine, and compared with the method for classifying texts through manual work, the method is high in accuracy.

In the 90 s of the 20 th century, improvements in a technology could clearly distinguish between improvements in hardware (e.g., improvements in circuit structures such as diodes, transistors, switches, etc.) and improvements in software (improvements in process flow). However, as technology advances, many of today's process flow improvements have been seen as direct improvements in hardware circuit architecture. Designers almost always obtain a corresponding hardware circuit structure by programming an improved method flow into the hardware circuit. Thus, it cannot be said that an improvement in the process flow cannot be realized by hardware physical modules. For example, a Programmable Logic Device (PLD) (e.g., a Field Programmable Gate Array (FPGA)) is an integrated circuit whose Logic functions are determined by a user programming the Device. A digital system is "integrated" on a PLD by the designer's own programming without requiring the chip manufacturer to design and fabricate application-specific integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing an Integrated Circuit chip, such Programming is often implemented by "logic compiler" software, which is similar to a software compiler used in program development, but the original code before compiling is also written in a specific Programming Language, which is called Hardware Description Language (HDL), and the HDL is not only one kind but many kinds, such as abll (Advanced boot Expression Language), AHDL (alternate hard Description Language), traffic, CUPL (computer universal Programming Language), HDCal (Java hard Description Language), lava, lola, HDL, PALASM, software, rhydl (Hardware Description Language), and vhul-Language (vhyg-Language), which is currently used in the field. It will also be apparent to those skilled in the art that hardware circuitry that implements the logical method flows can be readily obtained by merely slightly programming the method flows into an integrated circuit using the hardware description languages described above.

The controller may be implemented in any suitable manner, for example, the controller may take the form of, for example, a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, an Application Specific Integrated Circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include, but are not limited to, the following microcontrollers: ARC 625D, atmel AT91SAM, microchip PIC18F26K20, and Silicone Labs C8051F320, the memory controller may also be implemented as part of the control logic for the memory. Those skilled in the art will also appreciate that, in addition to implementing the controller as pure computer readable program code, the same functionality can be implemented by logically programming method steps such that the controller is in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Such a controller may thus be considered a hardware component, and the means included therein for performing the various functions may also be considered as a structure within the hardware component. Or even means for performing the functions may be regarded as being both a software module for performing the method and a structure within a hardware component.

The systems, devices, modules or units illustrated in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. One typical implementation device is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

For convenience of description, the above devices are described as being divided into various units by function, respectively. Of course, the functionality of the units may be implemented in one or more software and/or hardware when implementing the present application.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and so forth) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the specification. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instruction information. These computer program instruction information may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instruction information executed by the processor of the computer or other programmable data processing apparatus produce means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instruction information may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instruction information stored in the computer-readable memory produce an article of manufacture including instruction information means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instruction information may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instruction information executed on the computer or other programmable apparatus provides steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include forms of volatile memory in a computer readable medium, random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instruction information, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising a … …" does not exclude the presence of another identical element in a process, method, article, or apparatus that comprises the element.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The application may be described in the general context of computer-executable instruction information, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.

The above description is only an example of the present application and is not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method of determining a text category, the method comprising:

determining a category corresponding to each target text in the target text set according to a set text classification algorithm based on each text similarity value and the plurality of target texts subjected to category labeling;

determining a category corresponding to each target text in the target text set according to a set text classification algorithm based on each text similarity value and the plurality of target texts subjected to category labeling, including:

for each target text in the target text set, determining at least one similar target text corresponding to the target text based on each text similarity value;

and according to similar target texts which are subjected to class labeling in the similar target texts corresponding to the target text, determining the class corresponding to the target text or updating the class corresponding to the target text at present by using a k-nearest neighbor KNN classification algorithm.

2. The method as claimed in claim 1, wherein determining a category corresponding to each target text in the target text set according to a set text classification algorithm based on each of the text similarity values and the plurality of target texts with category labels includes:

determining a category corresponding to each target text in the target texts which are not subjected to category marking according to a set text classification algorithm based on the text similarity values and the target texts which are subjected to category marking;

and the number of the first and second groups,

3. The method of claim 1, wherein said determining, for each of the target texts in the target text set, at least one similar target text corresponding to the target text based on the respective text similarity value comprises:

for each target text in the target text set, screening text similarity values meeting set conditions from the text similarity values of the target text and other target texts; the set condition comprises that the text similarity value is larger than or equal to a set threshold value, or the text similarity value is ranked in the top N according to the numerical value from large to small; wherein N is a positive integer;

4. The method of claim 1, before calculating, for each of the subsets of target text, a text similarity value between any two of the target texts in the subset of target text, the method further comprising:

and/or the presence of a gas in the gas,

splitting the target text subsets with the number of target texts being larger than or equal to a set number into at least two target text subsets; and splitting the target texts in each target text subset into target texts, wherein the target texts in each split target text subset comprise at least one other common word except the word corresponding to the target text subset before splitting.

5. The method of claim 1, prior to said determining the target subset of text to which each word in the target set of text corresponds, the method further comprising:

preprocessing each target text in the target text set;

wherein the pre-processing comprises at least one of:

6. An apparatus for determining a text category, comprising:

a second determining module, configured to determine, according to a set text classification algorithm, a category corresponding to each target text in the target text set based on each text similarity value and the plurality of target texts subjected to category labeling;

wherein the second determining module comprises:

a second determining unit, configured to determine, for each target text in the target text set, at least one similar target text corresponding to the target text based on the text similarity value;

and the execution unit is used for determining the category corresponding to the target text or updating the category corresponding to the target text at present by using a k nearest neighbor KNN classification algorithm according to the similar target texts which are subjected to category marking in the similar target texts corresponding to the target text.

7. The apparatus of claim 6, the second determination module, comprising:

a first determining unit, configured to determine, according to a set text classification algorithm, a category corresponding to each target text in the plurality of target texts without performing category labeling based on each text similarity value and the plurality of target texts with performed category labeling;

and (c) a second step of,

and the updating unit is used for updating the category corresponding to each target text in the target texts subjected to category marking according to a set text classification algorithm based on the text similarity values and the target texts subjected to category marking.

8. The apparatus of claim 6, the second determination unit, comprising:

a screening subunit, configured to screen, for each target text in the target text set, a text similarity value that meets a set condition from the text similarity values of the target text and other target texts; the set condition comprises that the text similarity value is larger than or equal to a set threshold value, or the text similarity value is ranked in the top N from large to small according to the numerical value; wherein N is a positive integer;

9. The apparatus of claim 6, further comprising:

a deleting module, configured to delete a target text subset corresponding to the words that are not related to the scene;

and/or the presence of a gas in the atmosphere,

the splitting module is used for splitting the target text subsets, the number of the target texts of which is greater than or equal to a set number, into at least two target text subsets; and the target texts in each split target text subset comprise at least one other common word except the word corresponding to the target text subset before splitting.

10. The apparatus of claim 6, further comprising:

wherein the pre-processing comprises at least one of:

11. An apparatus for determining a text category, comprising:

a processor; and

and according to the similar target texts with the class labels in the similar target texts corresponding to the target text, determining the class corresponding to the target text or updating the class corresponding to the target text at present by using a k-nearest neighbor KNN classification algorithm.

12. A storage medium storing computer-executable instructions that, when executed, implement the following: