CN109902181B - Text detection method and device - Google Patents
Text detection method and device Download PDFInfo
- Publication number
- CN109902181B CN109902181B CN201910163128.2A CN201910163128A CN109902181B CN 109902181 B CN109902181 B CN 109902181B CN 201910163128 A CN201910163128 A CN 201910163128A CN 109902181 B CN109902181 B CN 109902181B
- Authority
- CN
- China
- Prior art keywords
- text
- distribution
- topic
- distribution matrix
- theme
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 56
- 239000011159 matrix material Substances 0.000 claims abstract description 144
- 238000007635 classification algorithm Methods 0.000 claims abstract description 13
- 238000000034 method Methods 0.000 claims description 24
- 238000004364 calculation method Methods 0.000 claims description 7
- 238000012545 processing Methods 0.000 claims description 6
- 238000005070 sampling Methods 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 7
- 238000012706 support-vector machine Methods 0.000 description 4
- 238000003066 decision tree Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005065 mining Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000004880 explosion Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a text detection method and a text detection device, which relate to the technical field of text classification and can calculate a topic distribution matrix corresponding to a plurality of characteristic attributes and a word distribution matrix corresponding to a text topic when the plurality of characteristic attributes and the text topic of a text to be detected are obtained; generating a joint distribution matrix based on the plurality of topic distribution matrices; and performing class detection on the text to be detected by using a classification algorithm according to the joint distribution matrix and the word distribution matrix. By fusing a plurality of topic distribution matrixes, the problem that the change and the fuzziness of the topic clusters cannot be truly reflected by a single document-topic matrix can be effectively avoided, and the class detection of the document is improved.
Description
Technical Field
The invention relates to the technical field of text classification, in particular to a text detection method and a text detection device.
Background
Nowadays, the era of information explosion is in an era, wherein the amount of information represented by texts is extremely rapidly increased, including text information such as media information, technical reports, books, mails, microblogs, comments and the like, and how to mine useful subject information from a large amount of texts is a current primary task.
In recent years, LDA (document topic Allocation) has been widely applied in the fields of topic mining, information analysis, knowledge service, and the like, and mainly focuses on research directions such as hot topic discovery, emerging topic detection, academic evaluation, and the like. However, the existing LDA three-layer topic model can not truly reflect the change and ambiguity of topic clusters because of a single document-topic matrix, so that the document representation performance is limited, and the classification detection of texts is reduced.
Disclosure of Invention
In view of the above, an object of the present invention is to provide a text detection method and apparatus, so as to alleviate the technical problem that the document representation performance is limited because a single document-theme matrix of the existing LDA three-layer theme model cannot truly reflect changes and fuzziness of a theme cluster.
In a first aspect, an embodiment of the present invention provides a text detection method, where the method includes: acquiring a plurality of characteristic attributes and text topics of a text to be detected, wherein the characteristic attributes are characteristic attributes matched with the text topics; calculating a theme distribution matrix corresponding to the plurality of characteristic attributes and a word distribution matrix corresponding to the text theme; generating a joint distribution matrix based on the plurality of topic distribution matrices; and performing class detection on the text to be detected by using a classification algorithm according to the joint distribution matrix and the word distribution matrix.
With reference to the first aspect, an embodiment of the present invention provides a first possible implementation manner of the first aspect, where the step of calculating a topic distribution matrix corresponding to a plurality of feature attributes and a word distribution matrix corresponding to a text topic includes: calculating the number of themes corresponding to the characteristic attribute by using the text confusion degree to acquire the hyper-parameter corresponding to the characteristic attribute; and respectively inputting the hyper-parameters corresponding to the characteristic attributes and preset hyper-parameters corresponding to the text topics into a pre-established LDA model to obtain a topic distribution matrix corresponding to the characteristic attributes and a word distribution matrix corresponding to the text topics.
With reference to the first possible implementation manner of the first aspect, an embodiment of the present invention provides a second possible implementation manner of the first aspect, where the step of calculating the number of topics corresponding to the feature attribute by using text obfuscation includes: presetting a plurality of theme numbers corresponding to the characteristic attributes; respectively calculating a confusion value corresponding to each topic number based on the text confusion degree; and selecting the theme number corresponding to the minimum confusion value as the theme number corresponding to the characteristic attribute.
With reference to the first possible implementation manner of the first aspect, an embodiment of the present invention provides a third possible implementation manner of the first aspect, where after calculating a topic distribution matrix corresponding to a plurality of feature attributes and a word distribution matrix corresponding to a text topic, the method further includes: judging whether the topic distribution matrixes corresponding to the characteristic attributes and the word distribution matrixes corresponding to the text topics obey Dirichlet distribution or not; if not, recalculating the hyper-parameters corresponding to the characteristic attributes not complying with the Dirichlet distribution or reselecting the preset hyper-parameters corresponding to the text topics not complying with the Dirichlet distribution.
With reference to the third possible implementation manner of the first aspect, an embodiment of the present invention provides a fourth possible implementation manner of the first aspect, where the step of generating a joint distribution matrix based on the topic distribution matrices corresponding to the multiple feature attributes includes: calculating probability matrixes of the theme distribution matrixes corresponding to the characteristic attributes on the theme respectively according to the Gibbs sampling; and processing the probability matrix by utilizing the Hadamard sum and the projection residual joint vector, and fusing the processed probability matrix to generate a joint distribution matrix.
With reference to the fourth possible implementation manner of the first aspect, an embodiment of the present invention provides a fifth possible implementation manner of the first aspect, where the step of performing category detection on the text to be detected by using a classification algorithm according to the joint distribution matrix and the word distribution matrix includes: generating a feature word distribution matrix based on the joint distribution matrix and the word distribution matrix; judging whether the feature word distribution matrix obeys polynomial distribution or not; and if so, inputting the feature word distribution matrix into a pre-trained text classifier so as to perform class detection on the text to be detected.
In a second aspect, an embodiment of the present invention further provides a device for detecting a text, where the device includes: the acquisition module is used for acquiring a plurality of characteristic attributes and text topics of the text to be detected, wherein the characteristic attributes are characteristic attributes matched with the text topics; the calculation module is used for calculating a theme distribution matrix corresponding to the characteristic attributes and a word distribution matrix corresponding to the text theme; a generation module for generating a joint distribution matrix based on the plurality of topic distribution matrices; and the detection module is used for carrying out class detection on the text to be detected by utilizing a classification algorithm according to the joint distribution matrix and the word distribution matrix.
With reference to the second aspect, an embodiment of the present invention provides a first possible implementation manner of the second aspect, where the computing module is further configured to: calculating the number of themes corresponding to the characteristic attribute by using the text confusion degree to acquire the hyper-parameter corresponding to the characteristic attribute; and respectively inputting the hyper-parameters corresponding to the characteristic attributes and preset hyper-parameters corresponding to the text topics into a pre-established LDA model to obtain a topic distribution matrix corresponding to the characteristic attributes and a word distribution matrix corresponding to the text topics.
With reference to the first possible implementation manner of the second aspect, an embodiment of the present invention provides a second possible implementation manner of the second aspect, where the calculation module is further configured to: presetting a plurality of theme numbers corresponding to the characteristic attributes; respectively calculating a confusion value corresponding to each topic number based on the text confusion degree; and selecting the theme number corresponding to the minimum confusion value as the theme number corresponding to the characteristic attribute.
With reference to the first possible implementation manner of the second aspect, an embodiment of the present invention provides a third possible implementation manner of the second aspect, where after the calculating module, the apparatus further includes: the judging module is used for judging whether the topic distribution matrixes corresponding to the characteristic attributes and the word distribution matrixes corresponding to the text topics obey Dirichlet distribution or not; and the recalculating module is used for recalculating the hyper-parameters corresponding to the characteristic attributes not complying with the Dirichlet distribution or reselecting the preset hyper-parameters corresponding to the text topics not complying with the Dirichlet distribution if the judging module judges that the text topics are not complying with the Dirichlet distribution.
The embodiment of the invention has the following beneficial effects:
the text detection method and the text detection device provided by the embodiment of the invention can calculate the topic distribution matrix corresponding to a plurality of characteristic attributes and the word distribution matrix corresponding to the text topic when acquiring a plurality of characteristic attributes and text topics of a text to be detected; generating a joint distribution matrix based on the plurality of topic distribution matrices; and performing class detection on the text to be detected by using a classification algorithm according to the joint distribution matrix and the word distribution matrix. By fusing a plurality of topic distribution matrixes, the problem that the change and the fuzziness of the topic clusters cannot be truly reflected by a single document-topic matrix can be effectively avoided, and the class detection of the document is improved.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and drawings.
In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a flowchart of a text detection method according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a process of detecting a network reimbursement text according to an embodiment of the present invention;
fig. 3 is a flowchart of another text detection method according to an embodiment of the present invention;
FIG. 4 is a diagram illustrating classification accuracy provided by an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a text detection apparatus according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of another text detection apparatus according to an embodiment of the present invention.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
LDA is a document theme generation model, comprising three layers of word, theme and document structure. The generative model is a process in which each word of an article is obtained by "selecting a topic with a certain probability and selecting a word from the topic with a certain probability". Document-to-topic follows a polynomial distribution, and topic-to-word follows a polynomial distribution. LDA is an unsupervised machine learning technique that can be used to identify underlying topic information in large-scale document sets or corpora. The method adopts a bag-of-words method, and each document is regarded as a word frequency vector, so that text information is converted into digital information which is easy to model.
At present, the LDA three-layer theme model can not truly reflect the change and the fuzziness of a theme cluster due to a single document-theme matrix, so that the document representation performance is limited. Based on this, the text detection method and apparatus provided by the embodiments of the present invention can improve the above technical problem.
For the convenience of understanding the embodiment, a detailed description will be given to a text detection method disclosed in the embodiment of the present invention.
The first embodiment is as follows:
an embodiment of the present invention provides a text detection method, such as a flowchart of a text detection method shown in fig. 1, where the method includes the following steps:
step S102, acquiring a plurality of characteristic attributes and text topics of a text to be detected, wherein the characteristic attributes are characteristic attributes matched with the text topics;
during specific implementation, the type of the text to be detected is a network marketing text, wherein the text subject content of the network marketing text is various, and the text subject content is often accompanied by current hot topics, such as virtual currency, charitable mutual assistance, internet + finance, national major strategic deployment and the like. In the embodiment of the invention, the "reimbursement" is taken as the text topic of the text mining of the network reimbursement, and the two characteristic attributes related to the information of the "reimbursement" text topic are respectively high interest and hierarchy remuneration by capturing and analyzing the text data of the network reimbursement.
Step S104, calculating a theme distribution matrix corresponding to a plurality of characteristic attributes and a word distribution matrix corresponding to a text theme;
step S106, generating a joint distribution matrix based on a plurality of theme distribution matrices;
and S108, performing category detection on the text to be detected by using a classification algorithm according to the joint distribution matrix and the word distribution matrix.
In actual use, the network reimbursement text needs to be preprocessed, each carriage return symbol in the text is taken as the text to be ended to obtain a paragraph corpus, the paragraph is taken as the minimum processing unit to detect the network reimbursement text, as shown in fig. 2, a schematic diagram of a network reimbursement text detection process is shown in fig. 2, and in the process of detecting the network reimbursement text, firstly, Gibbs sampling is utilized to estimate a dirichlet prior hyperparameter alpha corresponding to two characteristic attributes of 'interest higher degree' and 'hierarchy remuneration'1,α2Setting a Dirichlet prior hyper-parameter beta corresponding to the 'biography and reimbursement' text theme; then, alpha is adjusted1,α2And beta Dirichlet prior hyper-parameters are respectively input into a pre-trained LDA model to obtain topic distribution corresponding to two characteristic attributes of ' high interest ' and ' level compensationThe matrix delta, theta and a word distribution matrix phi corresponding to the text subject of the 'biography and marketing'; then, fusing the two characteristic attributes of 'high interest' and 'level remuneration' corresponding to the theme distribution matrix to generate a combined distribution matrix z; and finally, generating a feature word distribution matrix w according to the combined distribution matrix and the word distribution matrix, and inputting the feature word distribution matrix into a pre-trained text classifier to perform class detection on whether the network reimbursement text belongs to the text topic of 'reimbursement'.
The text detection method provided by the embodiment of the invention can calculate the topic distribution matrix corresponding to a plurality of characteristic attributes and the word distribution matrix corresponding to the text topic when acquiring a plurality of characteristic attributes and text topics of the text to be detected; generating a joint distribution matrix based on the plurality of topic distribution matrices; and performing class detection on the text to be detected by using a classification algorithm according to the joint distribution matrix and the word distribution matrix. By fusing a plurality of topic distribution matrixes, the problem that the change and the fuzziness of the topic clusters cannot be truly reflected by a single document-topic matrix can be effectively avoided, and the classification detection of the documents is improved.
Further, in order to better perform category detection on the text to be detected, a theme distribution matrix and a word distribution matrix with better quality need to be obtained, so that step S104 calculates a theme distribution matrix corresponding to a plurality of feature attributes and a word distribution matrix corresponding to a text theme, which can be realized by step S11 and step S12:
step 11, calculating the number of themes corresponding to the characteristic attribute by using the text confusion degree to acquire the hyper-parameters corresponding to the characteristic attribute;
in the implementation of the invention, the text confusion is used for determining the optimal number of the two characteristic attributes, namely high interest and hierarchy compensation. The following is a detailed description by taking the determination of the optimal number of topics of the "high-interest" feature attributes as an example: for example, the size of the number of topics corresponding to the "high-interest" feature attribute is set to be 10 to 100, 10 topic numbers of 10, 20 and 30 … 100 are respectively substituted into the text confusion calculation formula by taking 10 as a step length to calculate the confusion value corresponding to each topic number, and the topic number corresponding to the lowest confusion value in the 10 confusion values is selected as the optimal topic number of the "high-interest" feature attribute. The process of obtaining the optimal number of topics of the "hierarchy compensation" feature attribute is the same as the process of obtaining the optimal number of topics of the "interest" feature attribute, and the process of obtaining the optimal number of topics of the "hierarchy compensation" feature attribute is not described in the embodiment of the present invention.
Preferably, the hyper-parameters corresponding to the two characteristic attributes of "interest" and "hierarchy compensation" are selected by
This formula is calculated, where α represents the hyperparameter and k represents the number of topics. Table 1 shows the values of α for different topic numbers:
TABLE 1
| k | 10 | 20 | 50 | 100 | 200 | 300 | 400 | 500 |
| α | 5 | 2.5 | 1 | 0.5 | 0.25 | 0.17 | 0.13 | 0.1 |
Therefore, the value of the hyper-parameter alpha is inversely proportional to the number of the subjects, and the value of the alpha floats according to the difference of k, so that the optimal hyper-parameter can be obtained only by obtaining the optimal number of the subjects with characteristic attributes.
And step 12, respectively inputting the hyper-parameters corresponding to the characteristic attributes and preset hyper-parameters corresponding to the text topics into a pre-established LDA model to obtain topic distribution matrixes corresponding to the characteristic attributes and word distribution matrixes corresponding to the text topics.
In the concrete implementation, the setting of the dirichlet priori hyperparameter beta corresponding to the 'reimbursement' text subject is basically a fixed value, the setting is uniformly that beta is 0.01, and the dirichlet priori hyperparameter alpha corresponding to the two characteristic attributes of 'high interest' and 'hierarchy remuneration' is set1,α2And the Dirichlet prior hyper-parameter beta corresponding to the 'reimbursement' text theme is respectively input into a pre-established LDA model, and a theme distribution matrix corresponding to the two characteristic attributes of 'high interest' and 'level remuneration' and a word distribution matrix corresponding to the 'reimbursement' text theme are respectively obtained.
Further, based on the above process of the text detection method, fig. 3 shows a flowchart of another text detection method, which includes the following steps:
step S302, acquiring a plurality of characteristic attributes and text topics of a text to be detected, wherein the characteristic attributes are characteristic attributes matched with the text topics;
step S304, calculating a theme distribution matrix corresponding to a plurality of characteristic attributes and a word distribution matrix corresponding to a text theme;
step S306, judging whether the topic distribution matrixes corresponding to the characteristic attributes and the word distribution matrixes corresponding to the text topics obey Dirichlet distribution or not;
step S308, if not, recalculating the hyper-parameters corresponding to the characteristic attributes not complying with the Dirichlet distribution or reselecting the preset hyper-parameters corresponding to the text topics not complying with the Dirichlet distribution.
In concrete implementation, after a theme distribution matrix and a word distribution matrix are obtained, whether the matrixes are subjected to Dirichlet distribution or not needs to be judged, when the matrixes are subjected to Dirichlet distribution, the type of a network reimbursement text is detected, if the theme distribution matrix corresponding to 'high interest' and/or 'hierarchical rewarding' is judged not to be subjected to Dirichlet distribution, the hyper-parameters corresponding to the characteristic attributes of 'high interest' and/or 'hierarchical rewarding' need to be recalculated, and if the word distribution matrix corresponding to 'reimbursement' is judged not to be subjected to Dirichlet distribution, the hyper-parameters corresponding to 'reimbursement' need to be reset.
Step S310, generating a joint distribution matrix based on a plurality of theme distribution matrices;
and step S312, performing category detection on the text to be detected by using a classification algorithm according to the joint distribution matrix and the word distribution matrix.
In practical application, in order to overcome the technical problem that a single document-topic matrix cannot truly reflect the change and ambiguity of a topic cluster, so that the document representation performance is limited, in the embodiment of the present invention, topic distribution matrices corresponding to a plurality of feature attributes are fused to generate a joint distribution matrix, and therefore, step S106 generates the joint distribution matrix based on the topic distribution matrices corresponding to the plurality of feature attributes, which can be implemented by step 21 and step 22:
step 21, calculating probability matrixes of the theme distribution matrixes corresponding to the characteristic attributes on the theme respectively according to the Gibbs sampling;
and step 22, processing the probability matrix by utilizing the Hadamard sum and the projection residual joint vector, and fusing the processed probability matrix to generate a joint distribution matrix.
Specifically, a probability matrix P (z | delta) of the ith 'high-interest' topic distribution matrix and the jth 'hierarchical rewarding' topic distribution matrix on the 'reimbursement' text topic z is calculated according to Gibbs samplesi) And P (z | theta)j) (ii) a The probability matrix is then generalized using the hadamard product:
and simultaneously introducing a projection residual united vector which is used for measuring the different degrees between the two characteristics:1-(P(z|δi))1/2·(P(z|θj))1/2(ii) a The purpose of processing the probability matrix by using the hadamard sum and the projection residual joint vector is to eliminate common feature words in the theme distribution matrix corresponding to the two feature attributes, so that the joint distribution is more representative, for example, when the same feature word "income" appears in "high interest" and "hierarchy remuneration", the "income" cannot independently represent a certain feature attribute, and therefore, the same feature word is removed by using the method.
After the probability matrix is processed, the topic distribution matrix can be fused to obtain a joint distribution matrix:
wherein Z isijAnd a joint distribution matrix generated by the theme distribution matrix representing the ith 'interest' and the theme distribution matrix representing the jth 'level compensation'.
Further, the step of performing category detection on the text to be detected by using a classification algorithm according to the joint distribution matrix and the word distribution matrix comprises the following steps: generating a feature word distribution matrix based on the joint distribution matrix and the word distribution matrix; judging whether the feature word distribution matrix obeys polynomial distribution or not; and if so, inputting the feature word distribution matrix into a pre-trained text classifier so as to perform class detection on the text to be detected. Specifically, in order to accurately perform category detection on the network marketing texts, a suitable text classifier needs to be selected, in the embodiment of the present invention, a cross experiment is adopted to respectively compare the expression effects of different classifiers on the classification accuracy of the feature word distribution matrix, fig. 4 shows a schematic diagram of classification accuracy, and as shown in fig. 4, four classifiers which are currently widely used are selected in the embodiment and are respectively: k Nearest neighbors (K-Nearest Neighbor, kNN), Decision Trees (DT), Support Vector Machines (SVM), and Fast text. The feature word distribution matrixes are respectively input into the pre-trained classifiers to obtain the classification accuracy of the network marketing texts by different classifiers, as shown in fig. 4, Fast text is superior to other classifiers in classification accuracy, the classification accuracy reaches 86.25%, is 1.61% higher than that of an SVM, and is superior to the SVM in time and space complexity. Therefore, Fast text is adopted as a classifier, so that the detection efficiency of network marketing texts can be improved in time and space.
Example two:
on the basis of the foregoing embodiments, an embodiment of the present invention further provides a text detection apparatus, as shown in fig. 5, which includes:
an obtaining module 502, configured to obtain a plurality of feature attributes and a text theme of a text to be detected, where a feature attribute is a feature attribute matched with a text theme;
a calculating module 504, configured to calculate a topic distribution matrix corresponding to the multiple feature attributes and a word distribution matrix corresponding to the text topic;
a generating module 506, configured to generate a joint distribution matrix based on the plurality of topic distribution matrices;
and the detection module 508 is configured to perform category detection on the text to be detected by using a classification algorithm according to the joint distribution matrix and the word distribution matrix.
Further, the calculation module 504 is further configured to: calculating the number of themes corresponding to the characteristic attribute by using the text confusion degree to acquire the hyper-parameter corresponding to the characteristic attribute; and respectively inputting the hyper-parameters corresponding to the characteristic attributes and preset hyper-parameters corresponding to the text topics into a pre-established LDA model to obtain a topic distribution matrix corresponding to the characteristic attributes and a word distribution matrix corresponding to the text topics.
In actual use, the calculation module 504 is further configured to: presetting a plurality of theme numbers corresponding to the characteristic attributes; respectively calculating a confusion value corresponding to each topic number based on the text confusion degree; and selecting the theme number corresponding to the minimum confusion value as the theme number corresponding to the characteristic attribute.
On the basis of fig. 5, fig. 6 shows a schematic structural diagram of another text detection apparatus, which further includes, after the calculation module:
a determining module 602, configured to determine whether a topic distribution matrix corresponding to the multiple feature attributes and a word distribution matrix corresponding to the text topic obey dirichlet distribution;
and a recalculating module 604, configured to recalculate the hyper-parameter corresponding to the feature attribute not subject to the dirichlet distribution or reselect the preset hyper-parameter corresponding to the text topic not subject to the dirichlet distribution if the determining module determines that the text topic is not subject to the dirichlet distribution.
The text detection device provided by the embodiment of the invention has the same technical characteristics as the text detection method provided by the embodiment, so that the same technical problems can be solved, and the same technical effects can be achieved.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the system and the apparatus described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In addition, in the description of the embodiments of the present invention, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, e.g., as meaning either a fixed connection, a removable connection, or an integral connection; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meaning of the above terms in the present invention can be understood in specific cases for those skilled in the art.
In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings, and are only for convenience of description and simplicity of description, but do not indicate or imply that the device or element being referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus, should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
Finally, it should be noted that: although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art will understand that the following embodiments are merely illustrative of the present invention, and not restrictive, and the scope of the present invention is not limited thereto: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present invention, and they should be construed as being included therein. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (8)
1. A method for detecting text, the method comprising:
acquiring a plurality of characteristic attributes and a text theme of a text to be detected, wherein the characteristic attributes are matched with the text theme;
calculating a plurality of theme distribution matrixes corresponding to the characteristic attributes and word distribution matrixes corresponding to the text themes;
generating a joint distribution matrix based on a plurality of the topic distribution matrices;
performing category detection on the text to be detected by using a classification algorithm according to the combined distribution matrix and the word distribution matrix;
wherein the step of generating a joint distribution matrix based on the topic distribution matrices corresponding to the plurality of feature attributes comprises:
calculating probability matrixes of a plurality of theme distribution matrixes corresponding to the characteristic attributes on the theme according to the Gibbs sampling;
processing the probability matrix by utilizing a Hadamard sum and a projection residual joint vector, and fusing the processed probability matrix to generate the joint distribution matrix;
the step of utilizing a classification algorithm to perform class detection on the text to be detected according to the joint distribution matrix and the word distribution matrix comprises the following steps:
generating a feature word distribution matrix based on the joint distribution matrix and the word distribution matrix;
judging whether the feature word distribution matrix obeys polynomial distribution or not;
and if so, inputting the feature word distribution matrix into a pre-trained text classifier so as to perform class detection on the text to be detected.
2. The method of claim 1, wherein the step of calculating a plurality of topic distribution matrices corresponding to the feature attributes and a plurality of word distribution matrices corresponding to the text topics comprises:
calculating the number of themes corresponding to the characteristic attribute by using text confusion to acquire a hyper-parameter corresponding to the characteristic attribute;
and respectively inputting the hyper-parameters corresponding to the characteristic attributes and preset hyper-parameters corresponding to the text topics into a pre-established LDA model to obtain a topic distribution matrix corresponding to the characteristic attributes and a word distribution matrix corresponding to the text topics.
3. The method according to claim 2, wherein the step of calculating the number of topics corresponding to the feature attributes by using text obfuscation comprises:
presetting a plurality of theme numbers corresponding to the characteristic attributes;
respectively calculating a confusion value corresponding to each topic number based on the text confusion;
and selecting the theme number corresponding to the minimum confusion value as the theme number corresponding to the characteristic attribute.
4. The method of claim 2, wherein after calculating a plurality of topic distribution matrices corresponding to the feature attributes and a plurality of word distribution matrices corresponding to the text topics, the method further comprises:
judging whether the topic distribution matrixes corresponding to the characteristic attributes and the word distribution matrix corresponding to the text topic obey Dirichlet distribution or not;
if not, recalculating the hyper-parameter corresponding to the characteristic attribute not complying with the Dirichlet distribution or reselecting the preset hyper-parameter corresponding to the text theme not complying with the Dirichlet distribution.
5. An apparatus for detecting text, the apparatus comprising:
the acquisition module is used for acquiring a plurality of characteristic attributes and text topics of a text to be detected, wherein the characteristic attributes are characteristic attributes matched with the text topics;
the calculation module is used for calculating a plurality of theme distribution matrixes corresponding to the characteristic attributes and word distribution matrixes corresponding to the text themes;
a generating module for generating a joint distribution matrix based on a plurality of the topic distribution matrices;
the detection module is used for carrying out category detection on the text to be detected by utilizing a classification algorithm according to the joint distribution matrix and the word distribution matrix;
the generating module is further used for calculating probability matrixes of the theme distribution matrixes corresponding to the characteristic attributes on the theme respectively according to Gibbs sampling;
processing the probability matrix by utilizing a Hadamard sum and a projection residual joint vector, and fusing the processed probability matrix to generate the joint distribution matrix;
the detection module is further used for generating a feature word distribution matrix based on the joint distribution matrix and the word distribution matrix;
judging whether the feature word distribution matrix obeys polynomial distribution or not;
and if so, inputting the feature word distribution matrix into a pre-trained text classifier so as to perform class detection on the text to be detected.
6. The apparatus of claim 5, wherein the computing module is further configured to:
calculating the number of themes corresponding to the characteristic attribute by using text confusion to acquire a hyper-parameter corresponding to the characteristic attribute;
and respectively inputting the hyper-parameters corresponding to the characteristic attributes and preset hyper-parameters corresponding to the text topics into a pre-established LDA model to obtain a topic distribution matrix corresponding to the characteristic attributes and a word distribution matrix corresponding to the text topics.
7. The apparatus of claim 6, wherein the computing module is further configured to:
presetting a plurality of theme numbers corresponding to the characteristic attributes;
respectively calculating a confusion value corresponding to each topic number based on the text confusion;
and selecting the theme number corresponding to the minimum confusion value as the theme number corresponding to the characteristic attribute.
8. The apparatus of claim 6, wherein after the computing module, the apparatus further comprises:
the judging module is used for judging whether the topic distribution matrixes corresponding to the characteristic attributes and the word distribution matrix corresponding to the text topic obey Dirichlet distribution or not;
and the recalculating module is used for recalculating the hyper-parameter corresponding to the characteristic attribute not complying with the Dirichlet distribution or reselecting the preset hyper-parameter corresponding to the text theme not complying with the Dirichlet distribution if the judging module judges that the text theme is not complied with the Dirichlet distribution.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910163128.2A CN109902181B (en) | 2019-03-04 | 2019-03-04 | Text detection method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910163128.2A CN109902181B (en) | 2019-03-04 | 2019-03-04 | Text detection method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109902181A CN109902181A (en) | 2019-06-18 |
| CN109902181B true CN109902181B (en) | 2021-04-23 |
Family
ID=66946270
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910163128.2A Active CN109902181B (en) | 2019-03-04 | 2019-03-04 | Text detection method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109902181B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103714171A (en) * | 2013-12-31 | 2014-04-09 | 深圳先进技术研究院 | Document clustering method |
| CN107943824A (en) * | 2017-10-17 | 2018-04-20 | 广东广业开元科技有限公司 | A kind of big data news category method, system and device based on LDA |
| CN108228867A (en) * | 2018-01-15 | 2018-06-29 | 武汉大学 | A kind of theme collaborative filtering recommending method based on viewpoint enhancing |
-
2019
- 2019-03-04 CN CN201910163128.2A patent/CN109902181B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103714171A (en) * | 2013-12-31 | 2014-04-09 | 深圳先进技术研究院 | Document clustering method |
| CN107943824A (en) * | 2017-10-17 | 2018-04-20 | 广东广业开元科技有限公司 | A kind of big data news category method, system and device based on LDA |
| CN108228867A (en) * | 2018-01-15 | 2018-06-29 | 武汉大学 | A kind of theme collaborative filtering recommending method based on viewpoint enhancing |
Non-Patent Citations (3)
| Title |
|---|
| Conditional Topical Coding: An Efficient Topic Model on rich features;Jun Zhu，Ni Lao，Ning Chen，Eric P. Xing;《ACM sigkdd international Conference on Knowledge Discovery & Data Mining》;20110821;全文 * |
| Text classification method based on self-training and LDA topic models;Miha Pavlinek, Vili Podgorelec;《Expert Systems With Applications》;20170308;全文 * |
| 面向微博的短文本多分类研究;葛文镇;《中国优秀硕士论文全文数据库》;20150616;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109902181A (en) | 2019-06-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Liu et al. | Jointly embedding multiple single-cell omics measurements | |
| CN111062259B (en) | Table identification method and apparatus | |
| US10915701B2 (en) | Caption association techniques | |
| CN112711953B (en) | Text multi-label classification method and system based on attention mechanism and GCN | |
| CN107430625B (en) | Classifying documents by clustering | |
| EP2431918B1 (en) | Graph lattice method for image clustering, classification, and repeated structure finding | |
| TW201909112A (en) | Image feature acquisition | |
| US10409886B2 (en) | Relatedness determination device, non-transitory tangible computer-readable medium for the same, and relatedness determination method | |
| CN108304371B (en) | Method and device for mining hot content, computer equipment and storage medium | |
| CN108052505A (en) | Text emotion analysis method and device, storage medium, terminal | |
| CN106778878B (en) | Character relation classification method and device | |
| CN109086355B (en) | Hot-spot association relation analysis method and system based on news subject term | |
| CN113807073B (en) | Text content anomaly detection method, device and storage medium | |
| Shen et al. | Node2vec representation for clustering journals and as a possible measure of diversity | |
| CN111597943B (en) | Table structure identification method based on graph neural network | |
| CN111506785A (en) | Network public opinion topic identification method and system based on social text | |
| WO2016200408A1 (en) | Hybrid classification system | |
| CN111178196A (en) | Method, device and equipment for cell classification | |
| CN109902181B (en) | Text detection method and device | |
| CN111723206B (en) | Text classification method, apparatus, computer device and storage medium | |
| US20220139069A1 (en) | Information processing system, information processing method, and recording medium | |
| CN111488400B (en) | Data classification method, device and computer readable storage medium | |
| Chen et al. | Low-error data recovery based on collaborative filtering with nonlinear inequality constraints for manufacturing processes | |
| JP2015138462A (en) | Relevancy determination apparatus, relevancy determination program and relevancy determination method | |
| CN111786999A (en) | Intrusion behavior detection method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20240411 Address after: 101200 room 205-211526, No. 40, Fuqian West Street, Pinggu town, Pinggu District, Beijing (cluster registration) Patentee after: BEIJING YONGBO TECHNOLOGY CO.,LTD. Country or region after: China Address before: 100000 No. 100 Chaoyang District Ping Tian Park, Beijing Patentee before: Beijing University of Technology Country or region before: China |