CN114756656A - Mining method of association rules for hydraulic engineering safety hazard description based on improved Apriori algorithm - Google Patents

Abstract

The invention discloses an association rule mining method for the description of potential safety hazards of water conservancy projects based on an improved Apriori algorithm. First, Jieba word segmentation is used to preprocess a large number of texts describing potential safety hazards of water conservancy projects of different categories, and the TextRank algorithm is used to extract keywords and analyze the results of word segmentation. Part-of-speech screening to obtain the keywords of the hidden danger description text; then based on the one-hot encoding, the keyword itemset is transformed into a Boolean matrix, the Apriori association rules are integrated and improved, the frequent itemsets in various hidden dangers are mined, and the association rules are analyzed; finally, the confidence level is used. In order to evaluate the value, the strong correlation between the descriptions of hidden dangers is mined, and the trend of hidden dangers is predicted. The invention performs data preprocessing for a large number of unstructured hidden danger descriptions, improves the traditional Apriori algorithm, improves the computing efficiency, can timely analyze the description of the hidden dangers of water conservancy projects, and is helpful for the prediction, investigation and management of hidden dangers of water conservancy projects.

Description

Mining method of association rules for hydraulic engineering safety hazard description based on improved Apriori algorithm

technical field

The invention relates to the technical field of investigation of potential safety hazards in water conservancy projects, in particular to a mining method for association rules for the description of potential safety hazards in water conservancy projects based on an improved Apriori algorithm.

Background technique

Water conservancy projects are multi-faceted and wide-ranging, and there are a large number of hidden dangers. There is often correlation between hidden dangers. It is of great significance to explore the correlation between various safety hazards. As the most classic association rule mining algorithm, Apriori algorithm has been applied in the field of hydraulic engineering. However, the traditional Apriori algorithm has technical problems of low efficiency and time-consuming for large data sets, and it is difficult to accurately analyze unstructured texts such as descriptions of security risks.

In order to solve the above problems, for example, Huang Liming et al. in "Data Mining and Analysis of Water Conservancy Project Construction Quality and Safety Supervision Based on Association Rules", according to the problem description, the technical problems existing in the project are divided into 50 problem categories, and the Apriori algorithm is used to analyze the problems. Engineering attributes and problem descriptions are used for association rule mining. This method preprocesses the original hidden danger data by merging and classifying them, and realizes the structured expression of the hidden danger description. It consumes a lot of time, and has a certain degree of subjectivity, which affects the accuracy of the results; another example is the statement in "Association Rule Mining of Hidden Safety Hazards in Hydropower Engineering Construction", which proposes an optimization algorithm for association rules based on phrase extraction, based on phrase extraction technology. The text is extracted for key phrases, and the phrase measurement value is used as the evaluation index to optimize the support in the Apriori algorithm, so as to mine the association rules between hidden attributes. This method uses phrase extraction technology to mine new words in the text and improve the quality of word segmentation. , at the same time redefine the support degree to make association rule mining more scientific, but the phrase extraction technology adopted by this method requires a lot of labor costs to build and maintain the required rule base, and the new words excavated and the proper nouns in the field of water conservancy engineering exist There is a certain deviation, and there are word segmentation results with no clear meaning, such as easy to cause, not set, etc., and there is no improvement in the iteration of the Apriori algorithm, and the operation efficiency is low.

Therefore, it is necessary to construct an efficient mining method of association rules for the description of potential safety hazards in water conservancy projects. Compared with the existing method, the method for mining association rules for the description of potential safety hazards in water conservancy projects constructed in the present invention based on the improved Apriori algorithm has the advantages of Two notable features: (1) Constructing a custom thesaurus in the field of hydraulic engineering to segment hidden texts, using TextRank algorithm for part-of-speech screening and keyword extraction, and converting the segmentation results into Boolean matrices based on one-hot encoding. The hidden text only retains some important keywords. (2) Optimize the Apriori algorithm. Before iterating layer by layer, perform a frequent itemset screening on the Boolean matrix. The pruning optimization is carried out in the iteration, and the data set is dynamically reduced at the same time, and the operation efficiency of association rule mining has been significantly improved.

SUMMARY OF THE INVENTION

The purpose of the present invention is to propose an association rule mining method based on the improved Apriori algorithm for the description of potential safety hazards in water conservancy projects, so as to solve the problems of difficulty in structurally expressing the description of hidden dangers in the safety construction of water conservancy projects and the low computing efficiency of the association rule mining algorithm, so as to excavate the construction of water conservancy projects in time. The association rules between hidden safety hazards in the water conservancy project provide auxiliary support for the prediction, investigation and management of safety hazards in water conservancy projects.

A method for mining association rules for the description of potential safety hazards in water conservancy projects based on the improved Apriori algorithm of the present invention includes the following steps:

Step S1, constructing a custom thesaurus in the field of water conservancy engineering, using jieba word segmentation to segment the hidden danger data set, removing stop words, filtering parts of speech through the TextRank algorithm, and using the extracted keywords as the input data set;

Step S2, step S2, based on one-hot one-hot encoding, the input data set obtained in step S1 is subjected to Boolean matrix conversion;

In step S3, the support degree of each item set is calculated by improving the Apriori algorithm, and the itemsets that are not less than the minimum support degree are screened out, which are frequent itemsets;

Step S4, perform confidence calculation on the obtained frequent itemsets, and filter out the association rules not less than the minimum confidence, that is, strong association rules A=>B, that is, when itemset A appears, itemset B will also be exhausted. may appear.

Preferably, the step S1 specifically includes:

Step S11, collecting entries in the field of water conservancy engineering, constructing a custom thesaurus in the field of water conservancy engineering, using jieba word segmentation, segmenting the hidden danger data set, and marking the part of speech;

Step S12, load the stop word database, and remove words that usually have no clear meaning themselves, such as modal particles, adverbs, prepositions, conjunctions, etc. in the word segmentation result;

Step S13, using the TextRank algorithm to sort the importance of word segmentation, and then extracting keywords whose parts of speech are nouns, verbs, gerunds, and place names in the word segmentation, as the final input data set.

After the above steps S11-S13, the hidden danger description texts that are difficult to be expressed in a structured way in the data set are preprocessed.

Preferably, the step S2 specifically includes:

Step S21, first convert the input data set D={D ₁ , D ₁ . . . , D _n } into a matrix T in the form of a DataFrame data frame, and the form of the matrix T is:

Among them, n is the number of hidden danger description data, m is the maximum number of word segmentations in the n data, T _ij is the j-th keyword of the i-th data, if it is empty, it is Null; D _i is the i-th item in the data set data, that is, the set {T _i1 ,T _i2 ,...,T _in };

Step S22, perform one-hot one-hot encoding on the data frame matrix T, and convert it into a Boolean matrix M; let I={I ₁ , I ₂ ,...,I _t } be the set composed of all different items in the data set D, The Boolean matrix M has the form:

Among them, t is the number of all different keywords in the data set D, M _ij is the Boolean value of the i-th data for the j-th keyword, if D _i contains I _j , that is, the i-th data contains the j-th key word, the value of _Mij is 1 (True), otherwise it is 0 (False).

The Chinese text preprocessed by step S1 is converted into an easily recognizable Boolean matrix through step S2.

Preferably, the step S3 specifically includes:

Step S31, before performing the iteration, perform a traversal on the Boolean matrix converted in step S22, count and sum up each column in the Boolean matrix, quickly mine 1-dimensional frequent itemsets, and delete non-frequent itemsets;

Step S32, during iteration, the k-1-dimensional frequent itemset set L _k-1 is connected to form a k-dimensional candidate item set set, denoted as C _k ; let I ₁ and I ₂ be the itemsets in L _k-1 , If k-2 items in them are the same, then I ₁ and I ₂ can be connected to generate a result item set, which is one of the candidate item set sets C _k ;

Step S33, prune the candidate item set C _k , where C _k is a superset of L _k ; for each item set T in L _k-1 , traverse all the candidate item sets in C _k in turn, and based on Boolean The matrix counts each candidate item set. After the traversal is complete, trim each candidate item set in C _k according to the count result. If the count is less than k, delete the candidate item set, and delete all itemsets that contain the item set in the dataset; otherwise, keep it and proceed to the next step. step;

Step S34, calculate the support degree for each candidate item set after pruning, and the support degree (support) formula is:

support(A=>B)=P(A∪B)

In the formula: A and B are the itemsets composed of any items in I, that is, the probability that the data contains each item in the itemsets A and B;

If the support degree of the candidate item set is not less than the minimum support degree, it is a k-frequent itemset, and is added to L _k , otherwise, the item set is deleted, and all itemsets containing this item set are deleted in the data set at the same time. After screening, go back to step S32 until no k-frequent itemsets are found.

Through steps 2 and S31, before the iteration, there is no need to calculate the support degree of each keyword, directly count and sum each column in the Boolean matrix, quickly mine 1-dimensional frequent itemsets, and delete non-frequent itemsets; based on Boolean The matrix can quickly filter a large number of data sets, thereby greatly reducing the data set, and improving the iterative operation efficiency in the case of a huge data set in the early stage of iteration. During iteration, the k-1-dimensional frequent itemset set L _k-1 is connected to form a k-dimensional candidate item set C _k , and for each item set T in L _k-1 , all candidate items in C _k are traversed in turn sets and counts each candidate item set based on a Boolean matrix. After the traversal is completed, each candidate item set in C _k is trimmed through the counting result. If the count is less than k, the candidate item set is deleted, and all itemsets containing this item set are deleted in the dataset at the same time. By optimizing the pruning link in step S32, the time complexity is reduced from O(n ² ) to O(n), and the overall iterative operation efficiency is improved.

In the traditional Apriori algorithm, for each k-1 dimensional subset of the candidate item set, L _k-1 needs to be searched again, which increases the operation time and affects the algorithm efficiency. The improved Apriori algorithm mainly lies in the rapid mining of frequent itemsets before the iteration, pruning and optimization in the iteration, and the dynamic reduction of the data set at the same time, thereby significantly improving the computational efficiency of the algorithm.

Compared with the prior art, the beneficial effects of the present invention are:

(1) In the present invention, by constructing a custom thesaurus in the field of water conservancy engineering, using the TextRank algorithm for keyword extraction and part-of-speech filtering, and through automatic word segmentation by machines, the preprocessing of the hidden danger description text that is difficult to structurally express is realized, and the existing The problems of high labor cost and inability to recognize the terminology of hydraulic engineering in the technology.

(2) Before the iteration, the present invention quickly screens a large number of data sets based on the Boolean matrix, thereby greatly reducing the data set and improving the iterative operation efficiency in the case of a huge data set in the early iteration; , which reduces the time complexity from O(n ² ) to O(n) and improves the overall iterative operation efficiency.

(3) The present invention greatly improves the operation efficiency of the Apriori algorithm by improving the algorithms before and during iteration, and overcomes the problem of low algorithm operation efficiency in the case of huge data sets in the prior art.

Description of drawings

Fig. 1 is the overall working flow chart of the present invention.

Detailed ways

The technical solutions of the present invention will be described in detail below with reference to the embodiments, but the protection scope of the present invention is not limited to the embodiments.

Example 1

As shown in Figure 1, the method of the present invention specifically comprises the following steps:

Step S1, construct a custom thesaurus in the field of water conservancy engineering, use jieba word segmentation to segment the hidden danger data set, remove stop words, filter the parts of speech through the TextRank algorithm, and use the extracted keywords as the input data set. The specific steps are:

Step S11, collect 1917 water conservancy project hidden danger description sentences as a sample data set, some of the sample data set sample data are shown in Table 1, collect 7480 water conservancy engineering field entries, build a custom thesaurus in the water conservancy engineering field, use jieba word segmentation, Segment the sample data and mark the part of speech.

Some entries such as: dark light, ground resistance value, dark application, dark pipe, dark culvert, dark hinge, dark room, dark beam, dark furnace piece, safety exit, safety exit light, safety exit light outlet, safety island, Safety valve debugging records, safety precautions, safety management, safety grounding, safety alarm system, etc.

Table 1

Step S12, load the stop word database, and remove the modal particles, adverbs, prepositions, conjunctions and other words that usually have no clear meaning in the word segmentation result, such as "@", "A", "and", "and" Wait.

Step S13, using the TextRank algorithm to sort the importance of word segmentation, and extracting keywords whose parts of speech are noun (n), verb (v), gerund (vn), and place name (ns) in the word segmentation.

After steps S11-S13, the final word segmentation results shown in Table 2 are obtained as the input data set.

Table 2 word segmentation results

In step S2, the Boolean matrix encoding is performed on the input data set obtained in step S1, that is, for each piece of data, if it contains the corresponding item set in the data set, it is marked as 1 (True), otherwise it is 0 (False). The specific steps are:

Step S21, first convert the input data set into a 1917*20 matrix in the form of a DataFrame data frame, wherein the number of rows is 1917, that is, the number of hidden danger description data, and the number of columns is 20, that is, the maximum number of word segmentations in the 1917 pieces of data, the i-th The word in the jth column of the row is the jth keyword of the ith data. If it is empty, it is Null. The DataFrame data frame is shown in Table 3.

Table 3DataFrame data frame

Step S22, perform one-hot encoding on the DataFrame data frame, and convert it into a Boolean matrix, the matrix is 1917*1139, wherein the number of rows is 1917, that is, the number of hidden danger description data, and the number of columns is 1139, that is, all different data in the data set. The number of keywords. If the i data contains the jth keyword, the value of the ith row and the jth column is 1 (True), otherwise it is 0 (False). The encoded Boolean matrix is shown in Table 4.

Table 4 Boolean matrix results

In step S3, the item set support is calculated by improving the Apriori algorithm, and the itemsets that are not less than the minimum support are selected as frequent itemsets. Since the data volume is as high as 1917, the minimum support threshold in this embodiment is set to 0.01. The frequent itemsets whose length is greater than or equal to 3 are filtered, and the results are shown in Table 5.

Table 5 Results of frequent itemsets

According to the results of frequent itemsets, during the safety construction process of the water conservancy project corresponding to this data set, there are often hidden safety hazards in the maintenance workshop of the mixing station, the power distribution room of the sand and gravel processing plant, etc. Safety hazards such as wearing by construction site personnel occur frequently.

In step S4, a confidence level is calculated on the obtained frequent itemsets, and an association rule not less than the minimum confidence level is screened out, which is a strong association rule. Use the association_rules() function to perform further association rule mining on the frequent itemsets obtained in step S3, and perform mining with confidence as the evaluation value. Set the minimum confidence level to 0.8. The results of strong association rules are shown in Table 6. A confidence level of 1 is an absolute correlation item.

Table 6 Results of strong association rules

The results of the strong association rule reflect that during the safe construction of the water conservancy project corresponding to this dataset, the scaffolding mesh net often falls off, and the personnel at the construction site often have problems with wearing it. The maintenance workshop of the mixing station is safe A place where hidden dangers occur more frequently.

In this embodiment, the beneficial effects of the present invention are embodied in: (1) for the 1917 pieces of data used in this embodiment, automatic word segmentation is realized through machine codes, and the word segmentation results are shown in Table 2, while in the aforementioned prior art Since the phrase extraction technology of the company did not introduce a custom thesaurus in the field of water conservancy engineering and did not perform part-of-speech filtering, words with no clear meaning and part of speech such as "easy to cause", "not required", and "not set" were generated, which reduced the number of word segmentations. quality, which affects the accuracy of association rule mining. (2) Based on the 1917 pieces of data used in this embodiment, code testing is performed through PyCharm. The traditional Apriori algorithm takes 0.1587965 seconds, while the improved Apriori algorithm proposed in the present invention only takes 0.0328939 seconds, which greatly improves the operational efficiency of the algorithm .

Claims (4)

1. The hydraulic engineering potential safety hazard description association rule mining method based on the improved Apriori algorithm is characterized by comprising the following steps of:

step S1, building a self-defined word bank in the hydraulic engineering field, carrying out word segmentation on the hidden danger data set by adopting jieba word segmentation, removing stop words, screening out the part of speech by a TextRank algorithm, and taking the extracted keywords as an input data set;

step S2, performing Boolean matrix conversion on the input data set obtained in the step S1 based on the one-hot code;

step S3, calculating the support degree of each item set by improving Apriori algorithm, and screening out frequent item sets not less than the minimum support degree;

and step S4, performing confidence calculation on the obtained frequent item set, and screening out a strong association rule not less than the minimum confidence.

2. The hydraulic engineering potential safety hazard description association rule mining method based on the improved Apriori algorithm as claimed in claim 1, wherein the step S1 specifically includes:

step S11, collecting entries in the hydraulic engineering field, constructing a self-defined word bank in the hydraulic engineering field, adopting jieba word segmentation, segmenting the hidden danger data set, and labeling the part of speech;

step S12, loading a stop word bank, and eliminating words which usually have no definite meaning per se, such as mood auxiliary words, adverbs, prepositions, conjunctions and the like in the word segmentation result;

and step S13, sorting the importance of the participles by adopting a TextRank algorithm, and extracting the keywords of which the parts of speech are nouns, verbs, vernouns and place names in the participles to serve as a final input data set.

3. The hydraulic engineering potential safety hazard description association rule mining method based on the improved Apriori algorithm as claimed in claim 2, wherein the step S2 specifically includes:

in step S21, the input data set D is first set to { D ═ D₁,D₁…,D_nAnd converting the matrix into a matrix T in a DataFrame data frame form, wherein the matrix T is in the form of:

wherein n is the number of the hidden danger description data, m is the maximum word segmentation number in n pieces of data, and T_ijThe j key word of the ith data is Null if the j key word is Null; d_iFor the ith data in the data set, i.e. the set { T }_i1,T_i2,…,T_in}；

Step S22, carrying out one-hot coding on the data frame matrix T, and converting the data frame matrix T into a Boolean matrix M; if I is ═ I₁,I₂,…,I_tThe boolean matrix M is a set of all the different entries in the dataset D, the form of the boolean matrix M being:

where t is the number of all different keywords in the dataset D, M_ijBoolean value for the ith data to the jth keyword, if D_iComprising I_jThat is, the ith data contains the jth keyword, then M_ijIs 1(True), otherwise is 0 (False).

4. The hydraulic engineering potential safety hazard description association rule mining method based on the improved Apriori algorithm as claimed in claim 3, wherein the step S3 specifically comprises:

step S31, before iteration, performing one-time traversal on the Boolean matrix obtained by conversion in the step S22, counting and summing each column in the Boolean matrix respectively, quickly mining a 1-dimensional frequent item set, and deleting a non-frequent item set;

step S32, while iterating, k-1 dimension frequent item set L_k-1Connected to form a k-dimensional candidate set denoted C_k(ii) a Let I₁And I₂Is L_k-1If k-2 items in the item set are the same, I₁、I₂Concatenating produces a set of result items, which is a set of candidate items C_kOne of them;

step S33, for candidate item set C_kPruning, C_kIs L_kA superset of (c); for L_k-1Each item set T in (1), sequentially traversing C_kAnd counting each candidate based on the boolean matrix. After traversing is finished, counting result pairs C_kCutting each candidate item set, if the count is less than k, deleting the candidate item set, and simultaneously deleting all item sets containing the item set in the data set; otherwise, reserving and carrying out the next step;

step S34, calculating a support degree (support) for each of the pruned candidate sets, where the support degree formula is:

support(A＝>B)＝P(A∪B)

in the formula: A. b is a set of items consisting of any item in I, namely representing the probability that the data contains each item in the sets of items A and B;

if the support degree of the candidate item set is not less than the minimum support degree, the candidate item set is a k-dimensional frequent item set, and L is added_kOtherwise, deleting the item set and deleting all item sets containing the item set in the data set. And returning to the step S32 after the screening is finished until the k-dimensional frequent item set cannot be found.

Images