CN107451120B - A content conflict detection method and system for public text intelligence - Google Patents

Abstract

The invention discloses a content conflict detection method and system for publishing text information. The method includes: establishing a public text information data set; extracting keywords, and constructing a keyword co-occurrence matrix; performing binarization processing on the keyword co-occurrence matrix to establish a keyword co-occurrence network; and extracting the keyword co-occurrence network. components, and obtain a component data set; judge each component, determine whether there is content conflict, and determine the conflicting content, the method of the present invention uses correlation analysis to directly detect and judge the content in the public text information, without the need for The structured description and storage of public text data reduces the amount of calculation, and overcomes the technical defect of poor content conflict detection accuracy due to the inability to synchronize the update of structured knowledge base with the public text intelligence of very real-time big data. The content conflict detection of open text intelligence with big data characteristics is realized.

Description

A content conflict detection method and system for public text intelligence

technical field

The invention relates to the application field of public text information, in particular to a content conflict detection method and system for public text information.

Background technique

Open intelligence, also known as open source intelligence, refers to intelligence collected and mined from public media (such as newspapers/publications, the Internet, and self-media platforms, etc.). The content of intelligence is mainly unstructured data, including numbers, texts, and pictures. , video, etc.

Open text intelligence refers to intelligence data in text format collected and mined from public media (such as newspapers/publications, the Internet, and self-media platforms, etc.).

Content conflict refers to inconsistent or contradictory descriptions of the same subject features in the same problem situation.

Open text intelligence has comparative advantages such as low acquisition cost, wide data source channels, and good data real-time performance. At the same time, with the advancement of self-media technology and the popularization of the Internet, public text intelligence presents the characteristics of big data, that is, the amount of data is growing at an alarming rate, the data generation has multi-source characteristics, and the data dissemination process is multi-channel parallel and complex. etc., there are inevitably conflicting contents in the massive public text intelligence, which makes the analysis and utilization of public text intelligence difficult; and the conscious information misleading of potential competitors aggravates the seriousness of the problem. Therefore, the first step in the efficient and accurate application of open text intelligence is the detection and discovery of conflicting content.

Conflict content is a key factor restricting the quality of public text intelligence data. If potential conflict content is not detected and eliminated in a timely and effective manner, the results of big data analysis of public text intelligence will be unreliable and its application value will be reduced. Currently, content conflict detection for text data is mainly for small-scale and medium-scale data, and is mainly used to detect and discover metadata or structured data conflicts.

For example, Zhang Keren from the 20th Research Institute of China Electronics Technology Group Corporation proposed a method for detecting conflict of command contents in a network management and control system, which includes the following steps:

1. Count the mutually exclusive instructions of the network management and control system in terms of content;

2. Establish multiple mutually exclusive instruction sets, and each instruction in each mutually exclusive instruction set is mutually exclusive;

3. Set the command interval time threshold t;

4. Record the instructions received by the same device within the time interval of t. If there are 2 or more instructions in the same mutually exclusive instruction set, the instruction content conflict occurs; otherwise, there is no instruction content conflict.

For another example, Zhao Xiaofei and Huang Zhiqiu proposed a description logic-based CWM (Common Warehouse Metamodel, CWM for short) metadata conflict detection method, which includes the following steps:

1. Establish a description logic DL _id that supports identity constraints on concepts;

2. Apply the description logic DL _id to formalize the CWM metadata and establish a DL _id knowledge base;

3. Define the description logic query language requirement set;

4. According to the description logic query language requirements, establish a query language in the following format:

5. Apply nRQL to query the DL _id knowledge base to find content conflicts.

The existing methods are mainly oriented to small-scale and medium-scale data in terms of content conflict detection of text data, and their characteristics are mainly reflected in: (1) the key steps of the text data are first structured description and storage; (2) structured On the basis of the knowledge base, a reasoning mechanism for conflict detection is established, such as mutually exclusive instruction set, conflict query language, etc., and then content conflict detection is carried out. For the open text intelligence with the characteristics of big data, the existing methods have the following defects: (1) Under the background of the open text intelligence showing the characteristics of big data, the workload of structured description and storage of the open text intelligence data is extremely huge. , it will become very difficult; (2) The conflict detection and reasoning mechanism established on the basis of a structured knowledge base is rigid and lacks flexibility. Under the circumstance that the real-time nature of public text intelligence big data is very strong, the content of the established content The conflict detection and reasoning mechanism will be very easy to not adapt to the new problem situation; (3) The detected conflict content is at the micro level, that is, there are content conflicts in several (usually 2, and a small number) texts, which are difficult to present. The content conflict in the overall level of the big data set shows that the existing content conflict detection methods cannot realize the content conflict detection of public text intelligence with the characteristics of big data.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a content conflict detection method and system for open text intelligence in order to realize the content conflict detection of open text intelligence with the characteristics of big data.

For achieving the above object, the present invention provides the following scheme:

A content conflict detection method for public text intelligence, comprising the following steps:

Obtaining public text intelligence, and establishing a public text intelligence data set, wherein the public text intelligence data set includes a plurality of texts;

extracting the keywords of each text in the public text intelligence data set, and constructing a keyword co-occurrence matrix;

Binarizing the keyword co-occurrence matrix to obtain a binarized keyword co-occurrence matrix;

establishing a keyword co-occurrence network according to the binarized keyword co-occurrence matrix;

extracting components in the keyword co-occurrence network to obtain a component data set;

Judging each component in the component data set to determine whether there is a content conflict in the corresponding component; and when the judgment result is that there is a content conflict in the corresponding component, determine that the public text information data set exists according to the component with content conflict. conflicting text.

Optionally, extracting the keywords of each text in the public text intelligence data set, and constructing a keyword co-occurrence matrix, specifically includes:

Perform word segmentation on each piece of text in the public text intelligence data set to obtain the entry set of the piece of text;

Calculate the expected cross-information entropy of each entry in the entry set of the text;

According to the expected size of the cross-information entropy of each entry, the entries in the entry set of the text are sorted in descending order;

Extract the first k entries in the sorted entry set as the keywords of the text;

Establish a keyword set according to the keywords of each text in the text intelligence data set;

Count the number of times that any two keywords in the keyword set appear together in the same text;

According to the co-occurrence times of each two keywords in the same text, a keyword co-occurrence matrix is established.

Optionally, perform binarization processing on the keyword co-occurrence matrix to obtain a binarized keyword co-occurrence matrix, which specifically includes:

Replace the elements in the keyword co-occurrence matrix that are greater than or equal to the set threshold with 1;

Elements in the keyword co-occurrence matrix that are smaller than the set threshold are replaced with 0.

Optionally, extract the components in the keyword co-occurrence network to obtain a component data set, which specifically includes:

According to the principle of co-occurrence between keywords in the same component and no co-occurrence between keywords in different components, extract the components in the keyword co-occurrence network;

All components in the extracted keyword co-occurrence network are combined into a component dataset.

Optionally, each component in the component data set is judged to determine whether there is a content conflict in the corresponding component; and when the judgment result is that there is a content conflict in the corresponding component, the published text is determined according to the component with content conflict. Conflicting texts in intelligence datasets, including:

Judging each component in the component data set, and judging whether there is a content semantic conflict in the corresponding component;

When the judgment result is that there is content conflict in the corresponding component, the corresponding text in the public text information data set is searched according to the keyword with content semantic conflict in the component, and the conflicting text in the public text information data set is determined.

A content conflict detection system for public text intelligence, comprising:

The public text intelligence data set building module is used to obtain public text intelligence and establish public text intelligence data sets;

a keyword co-occurrence matrix building module, used for extracting the keywords of each text in the public text intelligence data set, and constructing a keyword co-occurrence matrix;

The binarization processing module is used for binarizing the keyword co-occurrence matrix to obtain a binarized keyword co-occurrence matrix;

a keyword co-occurrence network establishment module, configured to establish a keyword co-occurrence network according to the binarized keyword co-occurrence matrix;

a component extraction module for extracting components in the keyword co-occurrence network to obtain a component data set;

The conflict judgment module is used to judge each component in the component data set, and judge whether there is a content conflict in the corresponding component; and when the judgment result is that there is a content conflict in the corresponding component, determine the content conflict according to the component. Conflicting text exists in the public text intelligence dataset.

Optionally, the keyword co-occurrence matrix building module specifically includes:

an entry division submodule, which is used to segment each text in the public text intelligence data set to obtain the entry set of the text;

The expectation calculation submodule is used to calculate the expectation of the cross information entropy of each entry in the entry set of the text;

a sorting submodule, used for sorting the entries in the entry set of the text in descending order according to the expected size of the cross-information entropy of each entry;

The keyword extraction sub-module is used to extract the first k entries in the sorted entry set as the keywords of the text;

The keyword set establishment sub-module is used to establish a keyword set according to the keywords of each text in the text intelligence data set;

Co-occurrence statistics sub-module, used to count the co-occurrence times of any two keywords in the keyword set in the same text;

The keyword co-occurrence matrix establishment sub-module is used to establish a keyword co-occurrence matrix according to the number of co-occurrences of any two keywords in the same text.

Optionally, the binarization processing module specifically includes:

A submodule is set to 1, for replacing the elements in the keyword co-occurrence matrix that are greater than or equal to the set threshold with 1;

The sub-module of setting 0 is used to replace the elements in the keyword co-occurrence matrix that are smaller than the set threshold with 0.

Optionally, the component extraction module specifically includes:

The component extraction sub-module is used for extracting the components in the keyword co-occurrence network according to the principle of co-occurrence between keywords in the same component and no co-occurrence between keywords in different components;

The component dataset builds a submodule for combining all components in the extracted keyword co-occurrence network into a component dataset.

Optionally, the conflict judgment module specifically includes:

a conflict judgment submodule, used for judging each component in the component data set, and judging whether there is a content semantic conflict in the corresponding component;

The conflict content determination sub-module is used to retrieve the corresponding text in the public text information data set according to the keyword of the content semantic conflict in the corresponding component when the judgment result is that there is content conflict in the corresponding component, and determine the public text information The dataset has conflicting text.

According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

The invention discloses a content conflict detection method and system for public text information. First, public text information is obtained, and a public text information data set is established; then, keywords of each text in the public text information data set are extracted to construct a public text information data set. keyword co-occurrence matrix; binarize the keyword co-occurrence matrix to obtain a binarized keyword co-occurrence matrix; then, establish a keyword co-occurrence network according to the binarized keyword co-occurrence matrix ; Extract the components in the keyword co-occurrence network to obtain a component data set; finally, judge each component in the component data set to determine whether there is content conflict, and determine the conflicting content. The method of the present invention directly detects and judges the content in the public text information by using the correlation analysis, without the need for a structured knowledge base, and without the need for structured description and storage of the public text data, which reduces the amount of calculation and overcomes the problem of The update of the knowledge base cannot be synchronized with the open text intelligence of the real-time big data, which causes the technical defect of poor content conflict detection accuracy, and realizes the content conflict detection of the open text intelligence with the characteristics of big data.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings required in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some of the present invention. In the embodiments, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative labor.

FIG. 1 is a flowchart of a content conflict detection method for public text intelligence provided by the present invention.

FIG. 2 is a structural block diagram of a content conflict detection system for disclosing text information provided by the present invention.

Detailed ways

The purpose of the present invention is to provide a content conflict detection method and system for open text intelligence, so as to realize the content conflict detection of open text intelligence with big data characteristics.

In order to make the above objects, features and advantages of the present invention more clearly understood, the invention will be described in further detail below with reference to the accompanying drawings and specific embodiments.

As shown in FIG. 1 , the present invention provides a content conflict detection method for public text intelligence, comprising the following steps:

Step 101: Obtain public text information, and establish a public text information data set, where the public text information data set includes a plurality of texts; specifically, the public text information data set T is, T={t ₁ ,t ₂ ,... ,t _m ,...,t _M }, where t _m is the mth text in the public text intelligence data set T, and M represents the total number of texts in the public text intelligence data set T.

Step 102, extracting the keywords of each text in the public text information data set, and constructing a keyword co-occurrence matrix;

Step 103, performing binarization processing on the keyword co-occurrence matrix to obtain a binarized keyword co-occurrence matrix;

Step 104, establishing a keyword co-occurrence network according to the binarized keyword co-occurrence matrix; specifically, the process of establishing the keyword co-occurrence network is to set the value in the binarized keyword co-occurrence matrix to 1. The keywords corresponding to the elements are connected to obtain the keyword co-occurrence network;

Step 105, extracting the components in the keyword co-occurrence network to obtain a component data set;

Step 106: Judging each component in the component data set, and judging whether there is a content conflict in the corresponding component; and when the judgment result is that there is a content conflict in the corresponding component, determine the public text information according to the component with content conflict. The dataset has conflicting text.

Optionally, in step 102, the keywords of each text in the public text intelligence data set are extracted to construct a keyword co-occurrence matrix, which specifically includes:

表示第m条文本t_m的第l_m个词条， l_m＝1,2,…,L_m，L_m表示第m条文本t_m中词条的总数，例如，对文本“南京市长江大桥”进行分词，可以得到，{南京，市长，江大桥，南京市，长江，大桥，长江大桥}。Perform word segmentation on each text in the public text information data set to obtain the entry set of the text; specifically, perform word segmentation on the m-th text t _m , and obtain the entry set of the text as:

Represents the _lmth entry in the _mth text _tm , lm = 1,2,..., _Lm , _Lm represents the total number of entries in the _mth text tm, for example, for the text "Nanjing Yangtze River "Bridge" can be obtained by word segmentation, {Nanjing, Mayor, River Bridge, Nanjing City, Yangtze River, Bridge, Yangtze River Bridge}.

Calculate the expected cross-information entropy of each entry in the entry set of the text;

交叉信息熵的期望为：Specifically, counting terms

The expectation of cross information entropy is:

表示在出现词条

时文本t_m属于类别c_i的概率；p(c_i)表示文本t_m类别的概率分布；

反映了文本t_m类别的概率分布与在出现了词条

的情况下文本类别的概率分布之间的距离，其值越大，词条

对文本 t_m类别分布的影响也就越大。in,

Indicates that the term appears

is the probability that the text t _m belongs to the category c _i ; p( _ci ) represents the probability distribution of the text t _m category;

reflects the probability distribution of the text t _m categories and the occurrence of the term

The distance between the probability distributions of text categories in the case of

The greater the impact on the category distribution of text _tm .

Calculate the expectation of the cross-information entropy of each entry in the text t _m , and get the entry feature set:

According to the expected size of the cross-information entropy of each entry, the entries in the entry set of the text are sorted in descending order;

否则，k＝10；Extract the first k entries in the sorted entry set as the keywords of the text; specifically, for the m-th text t _m , if L _m ≤ 200, then

otherwise, k=10;

w₁,为第m条文本t_m中排序后的第一个词条，k_m为对第m条文本提取关键词时k的取值，每条文本的关键词集组成关键词集合为D＝D₁∪D₂∪…∪D_M＝{d₁,d₂,…,d_s}，其中，s为公开文本情报数据集的关键词集合D中关键词数目。According to the keywords of each text in the text intelligence data set, a keyword set is established; specifically, the m-th text t _m keyword set is

w ₁ , is the first entry after sorting in the m-th text t _m , k _m is the value of k when extracting keywords from the m-th text, and the keyword set of each text is composed of the keyword set D =D ₁ ∪D ₂ ∪… _∪DM ={d ₁ ,d ₂ ,…,d _s }, where s is the number of keywords in the keyword set D of the public text intelligence dataset.

Count the number of times that any two keywords in the keyword set appear together in the same text;

Establish a keyword co-occurrence matrix according to the number of co-occurrences of each two keywords in the same text;

Specifically, for a group of words (d _u , d _v ) in the set D, where u=1,2,...,s; v=1,2,...,s; u≠v; count them in the same text The number of occurrences is denoted as a _u,v , then a keyword co-occurrence matrix based on the keyword set D is obtained: A=(a _u,v ) _s×s .

Optionally, in step 103, binarize the keyword co-occurrence matrix to obtain a binarized keyword co-occurrence matrix, which specifically includes:

Replace the elements in the keyword co-occurrence matrix that are greater than or equal to the set threshold with 1;

Elements in the keyword co-occurrence matrix that are smaller than the set threshold are replaced with 0.

Specifically, based on the number of keywords s in the keyword set D, a threshold ε (ε>0, and an integer) is set, if a _u,v ≥ε, then a' _u,v =1, otherwise a' _{u , v} = 0. A binarized keyword co-occurrence matrix A'=(a' _u,v ) _s×s is obtained.

Optionally, in step 105, components in the keyword co-occurrence network are extracted to obtain a component data set, which specifically includes:

According to the principle that there is co-occurrence between keywords in the same component and there is no co-occurrence between keywords in different components, the components in the keyword co-occurrence network are extracted; The connected keywords are placed in the same component, and the unconnected keywords in the keyword co-occurrence network are placed in different components, where the i-th component is C _i ={d _i,1 ,d _{i, 2} ,…};

All components in the extracted keyword co-occurrence network are combined into a component data set, and the specific component data set is {C ₁ ,C ₂ ,...,C _i ,...}.

Optionally, in step 106, judge each component in the component data set, and judge whether there is a content conflict in the corresponding component; and when the judgment result is that there is a content conflict in the corresponding component, determine the component according to the component with content conflict. Conflicting texts described in public text intelligence datasets, including:

Judging each component in the component data set, and judging whether there is a content semantic conflict in the corresponding component;

When the judgment result is that there is content conflict in the corresponding component, the corresponding text in the public text information data set is searched according to the keyword with content semantic conflict in the component, and the conflicting text in the public text information data set is determined.

Specifically, manual interpretation is performed on each component C _i ={d _i,1 ,d _i,2 ,...} in turn, if there is content semantics between the keywords d _i,x and d _i,y (x≠y) If there is a conflict, search the corresponding text in the public text information dataset T={t ₁ ,t ₂ ,…,t _m ,…,t _M } according to the keywords d _i,x and d _i,y (x≠y), It is determined that there is conflicting content; otherwise, it is considered that there is no content conflict in the text corresponding to the keywords in the component C _i set.

As shown in Figure 2, the present invention also provides a content conflict detection system for disclosing text intelligence, including:

The public text intelligence data set establishment module 201 is used for obtaining public text intelligence and establishing a public text intelligence data set;

The keyword co-occurrence matrix building module 202 is used to extract the keywords of each text in the public text intelligence data set, and construct a keyword co-occurrence matrix;

The binarization processing module 203 is configured to perform binarization processing on the keyword co-occurrence matrix to obtain a binarized keyword co-occurrence matrix;

A keyword co-occurrence network establishing module 204, configured to establish a keyword co-occurrence network according to the binarized keyword co-occurrence matrix;

A component extraction module 205, configured to extract components in the keyword co-occurrence network to obtain a component data set;

The conflict judgment module 206 is used to judge each component in the component data set, and judge whether there is a content conflict in the corresponding component; and when the judgment result is that there is a content conflict in the corresponding component, determine the content conflict according to the component with the content conflict. Conflicting texts described in public text intelligence datasets.

Optionally, the keyword co-occurrence matrix building module 202 specifically includes:

an entry division submodule, which is used to segment each text in the public text intelligence data set to obtain the entry set of the text;

The expectation calculation submodule is used to calculate the expectation of the cross information entropy of each entry in the entry set of the text;

a sorting submodule, used for sorting the entries in the entry set of the text in descending order according to the expected size of the cross-information entropy of each entry;

The keyword extraction sub-module is used to extract the first k entries in the sorted entry set as the keywords of the text;

The keyword set establishment sub-module is used to establish a keyword set according to the keywords of each text in the text intelligence data set;

Co-occurrence statistics sub-module, used to count the co-occurrence times of any two keywords in the keyword set in the same text;

The keyword co-occurrence matrix establishment sub-module is used to establish a keyword co-occurrence matrix according to the number of co-occurrences of any two keywords in the same text.

Optionally, the binarization processing module 203 specifically includes:

A submodule is set to 1, for replacing the elements in the keyword co-occurrence matrix that are greater than or equal to the set threshold with 1;

The sub-module of setting 0 is used to replace the elements in the keyword co-occurrence matrix that are smaller than the set threshold with 0.

Optionally, the component extraction module 205 specifically includes:

The component extraction sub-module is used for extracting the components in the keyword co-occurrence network according to the principle of co-occurrence between keywords in the same component and no co-occurrence between keywords in different components;

The component dataset builds a submodule for combining all components in the extracted keyword co-occurrence network into a component dataset.

Optionally, the conflict judgment module 206 specifically includes:

a conflict judgment submodule, used for judging each component in the component data set, and judging whether there is a content semantic conflict in the corresponding component;

The conflict content determination sub-module is used to retrieve the corresponding text in the public text information data set according to the keyword of the content semantic conflict in the corresponding component when the judgment result is that there is content conflict in the corresponding component, and determine the public text information The dataset has conflicting text.

According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

The invention discloses a content conflict detection method and system for public text information. First, public text information is obtained, and a public text information data set is established; then, keywords of each text in the public text information data set are extracted to construct a public text information data set. keyword co-occurrence matrix; binarize the keyword co-occurrence matrix to obtain a binarized keyword co-occurrence matrix; then, establish a keyword co-occurrence network according to the binarized keyword co-occurrence matrix ; Extract the components in the keyword co-occurrence network to obtain a component data set; finally, judge each component in the component data set to determine whether there is content conflict, and determine the conflicting content. The method of the present invention directly detects and judges the content in the public text information by using the correlation analysis, without the need for a structured knowledge base, and without the need for structured description and storage of the public text data, which reduces the amount of calculation and overcomes the problem of The update of the knowledge base cannot be synchronized with the open text intelligence of the real-time big data, which causes the technical defect of poor content conflict detection accuracy, and realizes the content conflict detection of the open text intelligence with the characteristics of big data.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant part can be referred to the description of the method.

The principles and implementations of the invention are described herein by using specific examples. The descriptions of the above embodiments are only used to help understand the method and the core idea of the present invention, and the described embodiments are only a part of the embodiments of the present invention. , rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

Claims (4)

1. A content conflict detection method of open text information is characterized by comprising the following steps:

acquiring public text information, and establishing a public text information data set, wherein the public text information data set comprises a plurality of texts;

extracting keywords of each text in the public text information data set, and constructing a keyword co-occurrence matrix;

carrying out binarization processing on the keyword co-occurrence matrix to obtain a binarization keyword co-occurrence matrix;

establishing a keyword co-occurrence network according to the binarization keyword co-occurrence matrix;

extracting components in the keyword co-occurrence network to obtain a component data set;

judging each component in the component data set, and judging whether content conflict exists in the corresponding component; and when the judgment result is that the corresponding component has content conflict, determining the text of the public text information data set with conflict according to the component with content conflict; the method specifically comprises the following steps: judging each component in the component data set, and judging whether content semantic conflict exists in the corresponding component; when the judgment result is that the corresponding component has content conflict, searching the corresponding text in the public text information data set according to the keywords with content semantic conflict in the component, and determining the text with conflict in the public text information data set;

the method for extracting the keywords of each text in the public text information data set and constructing the keyword co-occurrence matrix specifically comprises the following steps: segmenting words of each text in the public text intelligence data set to obtain an entry set of the text; calculating the expectation of the cross information entropy of each entry in the entry set of the text; according to the expected size of the cross information entropy of each entry, sorting the entries in the entry set of the entry text in a descending order; extracting the first k entries in the ordered entry set as keywords of the text; establishing a keyword set according to keywords of each text in the text information data set; counting the times of common occurrence of any two keywords in the same text in the keyword set; establishing a keyword co-occurrence matrix according to the co-occurrence times of every two keywords in the same text;

extracting components in the keyword co-occurrence network to obtain a component data set, which specifically comprises the following steps: extracting components in the keyword co-occurrence network according to the principle that co-occurrence exists among keywords in the same component and no co-occurrence exists among keywords in different components; and combining all the components in the extracted keyword co-occurrence network into a component data set.

2. The method according to claim 1, wherein the binarizing processing is performed on the keyword co-occurrence matrix to obtain a binarized keyword co-occurrence matrix, and specifically comprises:

replacing elements which are larger than or equal to a set threshold value in the keyword co-occurrence matrix with 1;

and replacing elements smaller than the set threshold value in the keyword co-occurrence matrix with 0.

3. A content conflict detection system for open text intelligence, comprising:

the public text information data set establishing module is used for acquiring public text information and establishing a public text information data set;

the keyword co-occurrence matrix construction module is used for extracting keywords of each text in the public text information data set and constructing a keyword co-occurrence matrix;

the binarization processing module is used for carrying out binarization processing on the keyword co-occurrence matrix to obtain a binarization keyword co-occurrence matrix;

the keyword co-occurrence network establishing module is used for establishing a keyword co-occurrence network according to the binarization keyword co-occurrence matrix;

the component extraction module is used for extracting components in the keyword co-occurrence network to obtain a component data set;

the conflict judging module is used for judging each component in the component data set and judging whether content conflicts exist in the corresponding components; and when the judgment result is that the corresponding component has content conflict, determining the text of the public text information data set with conflict according to the component with content conflict; the conflict judgment module specifically comprises: the conflict judgment submodule is used for judging each component in the component data set and judging whether content semantic conflicts exist in the corresponding components; a conflict content determining submodule for searching the corresponding text in the public text information data set according to the keyword with content semantic conflict in the component and determining the text with conflict in the public text information data set when the judgment result is that the content conflict exists in the corresponding component;

the keyword co-occurrence matrix construction module specifically comprises: the entry division submodule is used for carrying out word segmentation on each text in the public text intelligence data set to obtain an entry set of the text; the expectation calculation submodule is used for calculating the expectation of the cross information entropy of each entry in the entry set of the text; the sequencing submodule is used for sequencing the entries in the entry set of the entry text in a descending order according to the expected size of the cross information entropy of each entry; the keyword extraction submodule is used for extracting the first k entries in the ordered entry set as keywords of the text; the keyword set establishing sub-module is used for establishing a keyword set according to the keywords of each text in the text information data set; the co-occurrence frequency counting submodule is used for counting the co-occurrence frequency of any two keywords in the keyword set in the same text; the keyword co-occurrence matrix establishing submodule is used for establishing a keyword co-occurrence matrix according to the co-occurrence times of any two keywords in the same text;

the component extraction module specifically comprises: the component extraction submodule is used for extracting components in the keyword co-occurrence network according to the principle that co-occurrence exists among the keywords in the same component and the co-occurrence does not exist among the keywords in different components; and the component data set establishing submodule is used for combining all the components in the extracted keyword co-occurrence network into a component data set.

4. The system according to claim 3, wherein the binarization processing module specifically comprises:

a 1 setting sub-module, configured to replace an element, which is greater than or equal to a set threshold, in the keyword co-occurrence matrix with 1;

and the 0 setting sub-module is used for replacing the elements smaller than the set threshold value in the keyword co-occurrence matrix with 0.

Images