CN116578613B - Data mining system for big data analysis - Google Patents
Data mining system for big data analysis Download PDFInfo
- Publication number
- CN116578613B CN116578613B CN202310855939.5A CN202310855939A CN116578613B CN 116578613 B CN116578613 B CN 116578613B CN 202310855939 A CN202310855939 A CN 202310855939A CN 116578613 B CN116578613 B CN 116578613B
- Authority
- CN
- China
- Prior art keywords
- vector
- government
- layer
- entity
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000007418 data mining Methods 0.000 title claims abstract description 39
- 238000007405 data analysis Methods 0.000 title claims abstract description 26
- 239000013598 vector Substances 0.000 claims abstract description 146
- 239000011159 matrix material Substances 0.000 claims abstract description 98
- 238000005070 sampling Methods 0.000 claims abstract description 50
- 238000010586 diagram Methods 0.000 claims abstract description 15
- 238000012545 processing Methods 0.000 claims abstract description 8
- 238000007781 pre-processing Methods 0.000 claims abstract description 5
- 230000006399 behavior Effects 0.000 claims description 20
- 230000006870 function Effects 0.000 claims description 15
- 238000000034 method Methods 0.000 claims description 10
- 230000004913 activation Effects 0.000 claims description 8
- 238000004364 calculation method Methods 0.000 claims description 7
- 239000000284 extract Substances 0.000 claims description 7
- 230000004927 fusion Effects 0.000 claims description 7
- 230000008520 organization Effects 0.000 claims description 5
- 238000011156 evaluation Methods 0.000 description 14
- 238000010801 machine learning Methods 0.000 description 2
- 238000003062 neural network model Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000013210 evaluation model Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2465—Query processing support for facilitating data mining operations in structured databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/26—Visual data mining; Browsing structured data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/285—Clustering or classification
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Fuzzy Systems (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of big data, and discloses a data mining system for big data analysis, which comprises the following components: the data preprocessing module is used for identifying entities contained in the government affair data based on the government affair data, generating a government affair data map based on the relation between the entities and generating entity vectors for the entities; the diagram generation module is used for generating a diagram matrix based on the government affair data map; the region generation module equally divides the graph matrix into N non-overlapping sub-regions; sequentially splicing the regional parameters of all the subregions to generate a subregion vector; the data processing module is used for inputting the graph matrix, the subarea vector and the entity vector into the data coding model and outputting classification labels related to the data mining targets; the invention makes up the association relation of the missing entities by learning the graph matrix and the sampling area parameters, and is suitable for the data mining processing of government big data.
Description
Technical Field
The invention relates to the technical field of big data, in particular to a data mining system for big data analysis.
Background
Government affair big data refers to mass data collected, integrated, analyzed and utilized by government departments. The government affair big data is characterized in that: 1. large scale. The government affair big data is managed by taking TB and PB as units, and the data scale is far more than that of common enterprises. 2. The data is complex and diverse. The government affair big data contains data with different formats, such as structured data, unstructured data, semi-structured data and the like. 3. Multidimensional association. The government affair big data has the characteristic of multidimensional association, and new knowledge and rules can be found by analyzing the association existing between different dimensions. Government departments from which government big data are sourced have complex organization relations, so that the association relation between government big data is found out by a rule-based method or a general machine learning method, and the goal is difficult to finish when the government departments are applied to specific data mining tasks.
Disclosure of Invention
The invention provides a data mining system for big data analysis, which solves the technical problem that the association relation between government big data in the related technology is found out by a rule-based method or a general machine learning method.
The invention provides a data mining system for big data analysis, comprising: the data preprocessing module is used for identifying entities contained in the government affair data based on the government affair data, generating a government affair data map based on the relation between the entities and generating entity vectors for the entities; the diagram generation module is used for generating a diagram matrix based on the government affair data map, wherein the value of an element in the diagram matrix is 0 or 1, and whether a relation exists between the entities is indicated; and the region generation module equally divides the graph matrix into N non-overlapping subareas, and the size of each subarea is M elements.
Extracting the row number of the elements of the lower left corner and the upper right corner of the divided subareas to generate an area parameter expressed asWherein->Column number of elements respectively representing lower left corner of sub-region,/->The number of columns and rows of elements respectively representing the upper right corner of the sub-region; sequentially splicing the regional parameters of all the subregions to generate a subregion vector; a data processing module for inputting the graph matrix, the sub-region vector, and the entity vector into a data encoding model, the data encoding model comprising: first linear layer, first sampling layer, firstA hidden layer and a first fully-connected layer.
The first linear layer is calculated as follows:wherein->Representing a sub-region vector,/->Representing a sample area vector, +.>Weight parameter representing the first linear layer, < ->Meaning rounded up or rounded up to an integer, < >>Representing an activation function, a sinusoidal function or a hyperbolic tangent function is selected.
The sampling area vector is input into a first sampling layer, the first sampling layer restores the sampling area vector into the area parameter of the sampling area, a corresponding sampling area is generated in the graph matrix, and empty elements in the graph matrix corresponding to the sampling area are filled with 1.
And inputting the graph matrix and the entity vector updated by the first sampling layer into a first hiding layer, wherein the first hiding layer updates the entity vector to obtain a weighted entity vector, and inputting the graph matrix and the weighted entity vector into a first full-connection layer, and outputting a classification label related to a data mining target by the first full-connection layer.
Further, the elements of the ith row and jth column of the graph matrix are represented as，/>A0 indicates that there is no relation between the i and j-th entities,/->A relation of 1 indicates that there is a relation between the i and j-th entities.
Further, if the graph matrix cannot be equally divided, edge filling is performed on the graph matrix, and after each edge filling is completed, the number of rows and columns of the graph matrix is increased by 1, and the edge filling is performed until the graph matrix can be equally divided.
Further, the definition of the region parameters of the sampling region is the same as the definition of the region parameters of the sub-region; if the region parameter of the sampling region is negative, it indicates that the size of the sampling region is 0.
Further, the calculation formula of the first hidden layer is as follows:wherein->And->Tensor matrix representing entity vector and weighted entity vector, respectively,>representing the sum of the diagram matrix and the identity matrix, +.>Representation->Degree matrix of->Weight matrix representing the first hidden layer, +.>Representing the ReLU activation function.
Further, the first hidden layer has a multi-layer structure, and the calculation formula is as follows:wherein->And->Respectively representing the output and input of the first hidden layer of the l+1 layer, when l=1 +.>Tensor matrix representing entity vectors, when l > 1 +.>Output of the first hidden layer representing the first layer,/->Representing the sum of the diagram matrix and the identity matrix, +.>Representation->Degree matrix of->Representing the weight matrix of the first hidden layer of the layer l+1.
Further, the first linear layer includes a plurality of input channels, each input channel including a graph matrix of a different physical arrangement order; the first sampling layer generates a corresponding sampling area on each graph matrix respectively and updates each input graph matrix; the first hidden layer has a channel corresponding to each graph matrix, each channel outputs a group of weighted entity vectors, and the weighted entity vectors of the channels are added and divided by the channel number to be used as the weighted entity vectors of the input full-connection layer.
Further, the entity included in the government data identification government data includes a government staff post name, a government agency built-in agency name, a government work task name, a policy file name, and the like.
And the second sampling layer is arranged in front of the first full-connection layer, extracts an entity from a record of one government work task of a government staff, extracts a weighted entity vector corresponding to the entity extracted from the record of the one government work task from the weighted entity vector to generate a first entity set, performs feature fusion on the first entity set, and then inputs the first full-connection layer, and a classification label output by the first full-connection layer is used for representing offline workload of the one government work task.
Further, the system further comprises an online evaluation module and a total workload evaluation module, wherein the online evaluation module collects online work behavior information of one government staff for one government work task and then generates online feature vectors representing online work behaviors.
The on-line work behavior information of one government affair task of a government affair staff comprises the number of pages related to the government affair task, the number of words input on the pages related to the government affair task and the policy file name entity related to the government affair task; and encoding the online work behavior information of one government work task including the number of browsed pages related to the government work task and the number of words input on the pages related to the government work task to obtain page features and word features, and splicing the page features, the word features and entity vectors of policy file name entities related to the government work task to obtain a first spliced vector.
Then inputting the first spliced vector into a multi-layer perceptron, wherein the classification label of the multi-layer perceptron is the on-line workload time; and extracting the characteristics of an output layer input into the multi-layer perceptron as an on-line characteristic vector.
And the total workload assessment module is used for splicing the characteristic vectors after the characteristic fusion of the on-line characteristic vectors and the first entity set to obtain a second spliced vector, then inputting the second spliced vector into a total workload assessment model, and outputting a grading or grading of the classification label representing the total workload by the total workload assessment model.
Further, the system also comprises an portrait vector module and a vector index module, wherein the portrait vector module extracts a weighted entity vector of a portrait object name output by the first hidden layer as a portrait vector of the portrait object; the entity included in the government data identified by the government data includes a photo image, a portrait object name, a person track, a policy file name, a government agency name, and a government personnel post name.
The first full-connection layer outputs two classification labels which are respectively corresponding to the record of illegal actions of the portrait object and the record of illegal actions of the portrait object.
The vector index module clusters the portrait vectors of all the generated portrait objects, takes the cluster center as a code vector, and maps the portrait vectors in the cluster where the code vector is located with the portrait vectors.
The invention has the beneficial effects that: according to the invention, the study of the graph matrix and the sampling area parameters is used for making up for the missing association relation between the entities, making up for the missing relation between the required data entities in a specific data mining task, and the higher accuracy performance can be achieved by a lower hidden layer number, so that the method is suitable for the data mining processing of government affair big data.
Drawings
FIG. 1 is a block diagram of a data mining system for big data analysis according to the present invention.
FIG. 2 is a block diagram of a data mining system for big data analysis according to the present invention.
FIG. 3 is a block diagram of a data mining system for big data analysis according to the present invention.
In the figure: the system comprises a data preprocessing module 101, a graph generating module 102, a region generating module 103, a data processing module 104, a total work amount evaluation module 201, an on-line evaluation module 202, an portrait vector module 301 and a vector indexing module 302.
Detailed Description
The subject matter described herein will now be discussed with reference to example embodiments. It is to be understood that these embodiments are merely discussed so that those skilled in the art may better understand and implement the subject matter described herein and that changes may be made in the function and arrangement of the elements discussed without departing from the scope of the disclosure herein. Various examples may omit, replace, or add various procedures or components as desired. In addition, features described with respect to some examples may be combined in other examples as well.
As shown in fig. 1, a data mining system for big data analysis, comprising: the data preprocessing module 101 identifies entities included in the government data based on the government data, generates a government data map based on a relationship between the entities, and generates entity vectors for the entities.
For government data of text type, the entity is identified by Named Entity Recognition (NER), and the method for generating the entity vector can be generated by encoding a word frequency-inverse document frequency model (TF-IDF), a document-vector model (Doc 2 vec), and the like.
Manual and self-help algorithms (Bootstrapping) and the like can be used to determine relationships between entities.
The graph generation module 102 generates a graph matrix based on the government affair data graph, wherein the value of an element in the graph matrix is 0 or 1, and the graph matrix indicates whether a relationship exists between entities.
Elements of the ith row and jth column of the graph matrix are represented as，/>A 0 indicates that there is no relationship between the i and j-th entities,a relation of 1 indicates that there is a relation between the i and j-th entities.
The region generation module 103 equally divides the graph matrix into N non-overlapping sub-regions, each sub-region having a size of m×m elements.
If the image matrix cannot be equally divided, edge filling is carried out on the image matrix, and the number of rows and columns of the image matrix is increased by 1 after each edge filling is finished, so that the image matrix can be equally divided.
Wherein R represents the total number of elements of the graph matrix, M is the number of columns of the sub-region, +.>Representing an upward rounding. N is a custom superparameter, and is generally 100.
Extracting the row number of the elements of the lower left corner and the upper right corner of the divided subareas to generate an area parameter expressed asWherein->Column number of elements respectively representing lower left corner of sub-region,/->The number of rows and columns of elements in the upper right corner of the sub-region are respectively indicated.
And sequentially splicing the regional parameters of all the subregions to generate a subregion vector. The splicing order is to splice the elements in the upper right corner from small to large.
The row and column numbers are the corresponding row and column positions of the elements in the graph matrix, for example, the row and column numbers of the elements in the first row and column are 1.
A data processing module 104 for inputting the graph matrix, the sub-region vector, and the entity vector into a data encoding model, the data encoding model comprising: a first linear layer, a first sampling layer, a first hidden layer, and a first fully connected layer.
The first linear layer is calculated as follows:wherein->Representing a sub-region vector,/->Indicating the sampling area directionQuantity (S)>Weight parameter representing the first linear layer, < ->Meaning rounded up or rounded up to an integer, < >>Representing an activation function, a sinusoidal function or a hyperbolic tangent function is selected.
Through the calculation of the first linear layer, a sampling region scaled on the basis of the sub-region is obtained.
The sampling area vector is input into a first sampling layer, the first sampling layer restores the sampling area vector into the area parameter of the sampling area, a corresponding sampling area is generated in the graph matrix, and empty elements in the graph matrix corresponding to the sampling area are filled with 1.
The definition of the regional parameters of the sampling region is the same as the definition of the regional parameters of the sub-region; if the region parameter of the sampling region is negative, it indicates that the size of the sampling region is 0.
And inputting the graph matrix and the entity vector updated by the first sampling layer into a first hiding layer, wherein the first hiding layer updates the entity vector to obtain a weighted entity vector, and inputting the graph matrix and the weighted entity vector into a first full-connection layer, and outputting a classification label related to a data mining target by the first full-connection layer.
If the graph matrix was edge-filled, the graph matrix entered into the first hidden layer deletes the edge-filled portion.
The calculation formula of the first hidden layer is as follows:wherein->And->Representing entity vectors and weighted entity vectors, respectivelyTensor matrix of>Representing the sum of the graph matrix and the identity matrix of the same size,/->Representation->Degree matrix of->Weight matrix representing the first hidden layer, +.>Representing the ReLU activation function.
Further, the first hidden layer has a multi-layer structure, and the calculation formula is as follows:wherein->And->Respectively representing the output and input of the first hidden layer of the l+1 layer, when l=1 +.>Tensor matrix representing entity vectors, when l > 1 +.>Output of the first hidden layer representing the first layer,/->Representing the sum of the graph matrix and the identity matrix of the same size,/->Representation->Degree matrix of->Weight matrix representing the first hidden layer of the layer l+1,/and>representing the ReLU activation function.
The row vector of the tensor matrix of entity vectors represents an entity vector, and the serial number of the row vector is consistent with the serial number of the entity in the graph matrix. The tensor matrix of the weighted entity vector is represented in the same way as the tensor matrix of the entity vector.
In a further embodiment, the first linear layer comprises a plurality of input channels, each input channel comprising a different physical arrangement of the graph matrix; in particular, the physical arrangement order of the graph matrix may be random or manually ordered.
The first sampling layer generates a corresponding sampling area on each graph matrix respectively and updates each input graph matrix; the first hidden layer has a channel corresponding to each graph matrix, each channel outputs a group of weighted entity vectors, and the weighted entity vectors of the channels are added and divided by the channel number to be used as the weighted entity vectors of the input full-connection layer.
The entity vector input per channel of the first hidden layer is uniform, but the graph matrix is different.
The data coding model is trained by adopting a training method of a neural network model.
According to the data mining system for big data analysis, the unknown relation discovery capability between entities is generated through learning the graph matrix and the sampling area parameters, the defect of the relation between the needed data entities in a specific data mining task is overcome, the model can achieve higher accuracy performance with lower hidden layer number, and the data mining system is suitable for data mining processing of government big data.
If there is an inconsistent entity vector dimension, more than one linear layer may be provided before the first hidden layer for mapping entity vectors to the same dimension.
The data mining system for big data analysis also comprises a database for storing government data, graph matrix and the like.
As shown in fig. 2, in a specific embodiment, a data mining system for big data analysis is applied to the measurement of workload of government staff, and the objective of data mining is to evaluate workload of government staff, where the government data identifies that entities included in the government data include a post name of government staff, a name of government organization, a name of an organization built in the government organization, a name of a government work task, a name of a policy file, and so on.
For example, the name of a government task is a annual report of an enterprise in a censored area, and the contents of such a government task can be easily obtained on a government platform which distributes the task through a system.
And the second sampling layer is arranged in front of the first full-connection layer, extracts an entity from a record of one government work task of a government staff, extracts a weighted entity vector corresponding to the entity extracted from the record of the one government work task from the weighted entity vector to generate a first entity set, performs feature fusion on the first entity set, and then inputs the first full-connection layer, and a classification label output by the first full-connection layer is used for representing offline workload of the one government work task.
One method of feature fusion for the first set of entities is:wherein->Represents the e-th weighted entity vector in the first set of entities, and K represents the total number of weighted entity vectors in the first set of entities.
For example, the class labels correspond to time values representing the time of work offline for the government work task; the time of the on-line work of the government work task is easy to count, and the time of the on-line work and the off-line work of the government work task can be used as the total time of the government work task.
One problem that the present system overcomes is that a government task may be performed by multiple government workers, taking into account the coordination between the government workers and the workflow between the government departments.
One problem with this embodiment is that it is misshapen to evaluate the workload of a government task only with total working time, and on this basis, an on-line evaluation module 202 and a total workload evaluation module 201 are added, wherein the on-line evaluation module 202 collects on-line work behavior information of one government task of a government staff, and then generates an on-line feature vector characterizing the on-line work behavior.
The total workload assessment module 201 performs feature fusion on the online feature vector and the first entity set to obtain a second spliced vector, and then inputs the second spliced vector into a total workload assessment model, and the total workload assessment model outputs a score or a rating of the classification label representing the total workload.
For example, the evaluation levels are classified into large, medium, and small, and correspond to the outputs of one total workload evaluation model, respectively.
The type of total workload assessment model is a neural network model.
The on-line work behavior information of one government affair task of a government affair staff comprises the number of pages related to the government affair task, the number of words input on the pages related to the government affair task, the policy file name entity related to the government affair task and the like; and encoding the online work behavior information of one government work task including the number of browsed pages related to the government work task and the number of words input on the pages related to the government work task to obtain page features and word features, and splicing the page features, the word features and entity vectors of policy file name entities related to the government work task to obtain a first spliced vector.
Then inputting the first spliced vector into a multi-layer perceptron (MLP), wherein the classification label of the multi-layer perceptron is on-line workload time; and extracting the characteristics of an output layer input into the multi-layer perceptron as an on-line characteristic vector.
Based on this embodiment, we need to evaluate the total evaluation of the workload of a government staff in a period of time, and provide a method that the type of the total workload evaluation module 201 is a cyclic neural network (RNN), the second spliced vector of the government work task performed by a government staff in a period of time is sequentially input into the total workload evaluation module 201 according to the time sequence, and the total workload evaluation module 201 outputs a classification label to represent the score or rating of the total workload of a government staff in a period of time after the last input. I.e. the classification labels are output in the last time step if the input is performed in time steps.
The purpose of the workload evaluation can be specifically to adjust the rest time of government staff or evaluate the performance of the government staff.
As shown in fig. 3, in another specific embodiment, a data mining system for big data analysis is applied to the creation of a portrait, where the traditional creation mode of a portrait in government aspect is that related data of a storage object is structured, and the structured storage easily causes the deletion of the portrait, and requires a large amount of manual assistance to search for corresponding data, and in this embodiment, the goal of data mining is vectorization of the portrait; the entity included in the government data identified by the government data includes a photo image, a portrait object name, a person track, a policy file name, a government agency name, a government personnel post name, and the like.
The entities of the character track are recorded in time-ordered addresses, so that entity vectors are obtained by adopting a semantic coding mode.
A data mining system for big data analysis includes a portrait vector module 301, the portrait vector module 301 extracting a weighted entity vector of a portrait object name output by a first hidden layer as a portrait vector of the portrait object.
The first full-connection layer outputs two classification labels which are respectively corresponding to the record of illegal behaviors of the portrait object and the record of illegal behaviors of the portrait object; the record of illegal behavior of the portrait object actually represents the result that the weighted entity vector of the portrait object is input to the output of the first full connection layer.
Generally, character track data recorded in government affair big data are all information related to specific behaviors of portrait objects, so that the establishment method of the portrait images can obtain vectorization representation of the behaviors and portrait features simultaneously, obtain the information of the portrait objects of the related portrait images through vectorization indexes, and evaluate the behavior types of target portrait.
The data mining system for big data analysis further comprises a vector index module 302, which clusters the portrait vectors of all the generated portrait objects, takes the cluster center as a code vector, and maps the portrait vectors in the cluster where the code vector is located with the portrait vectors.
And performing similarity calculation on the image vector and the code vector of the new image object generated after the big data is updated to match the code vector with the maximum similarity as a neighboring code vector, and extracting information of the image object corresponding to the image vector mapped by the neighboring code vector as an index result or directly taking the image vector as an index result.
The embodiment has been described above with reference to the embodiment, but the embodiment is not limited to the above-described specific implementation, which is only illustrative and not restrictive, and many forms can be made by those of ordinary skill in the art, given the benefit of this disclosure, are within the scope of this embodiment.
Claims (10)
1. A data mining system for big data analysis, comprising: the data preprocessing module is used for identifying entities contained in the government affair data based on the government affair data, generating a government affair data map based on the relation between the entities and generating entity vectors for the entities; the diagram generation module is used for generating a diagram matrix based on the government affair data map, wherein the value of an element in the diagram matrix is 0 or 1, and whether a relation exists between the entities is indicated; the region generation module equally divides the graph matrix into N non-overlapping subareas, and the size of each subarea is M elements;
extracting the row number of the elements of the lower left corner and the upper right corner of the divided subareas to generate an area parameter expressed asWherein->Column number of elements respectively representing lower left corner of sub-region,/->The number of columns and rows of elements respectively representing the upper right corner of the sub-region; sequentially splicing the regional parameters of all the subregions to generate a subregion vector; a data processing module for inputting the graph matrix, the sub-region vector, and the entity vector into a data encoding model, the data encoding model comprising: a first linear layer, a first sampling layer, a first hidden layer, and a first fully-connected layer;
the first linear layer is calculated as follows:wherein->Representing a sub-region vector,representing a sample area vector, +.>Weight parameter representing the first linear layer, < ->Meaning rounded up or rounded up to an integer, < >>Representing activation functions, selecting sine functions orA hyperbolic tangent function;
the sampling area vector is input into a first sampling layer, the first sampling layer restores the sampling area vector into the area parameter of the sampling area, a corresponding sampling area is generated in the graph matrix, and empty elements in the graph matrix corresponding to the sampling area are filled with 1;
and inputting the graph matrix and the entity vector updated by the first sampling layer into a first hiding layer, wherein the first hiding layer updates the entity vector to obtain a weighted entity vector, and inputting the graph matrix and the weighted entity vector into a first full-connection layer, and outputting a classification label related to a data mining target by the first full-connection layer.
2. A data mining system for big data analysis according to claim 1, wherein the elements of row i and column j of the graph matrix are represented as，/>A0 indicates that there is no relation between the i and j-th entities,/->A relation of 1 indicates that there is a relation between the i and j-th entities.
3. The data mining system for big data analysis of claim 1, wherein if the graph matrix is not equally divided, the graph matrix is edge-filled, and the number of rows and columns of the graph matrix is increased by 1 after each edge filling is completed, and the edge filling is performed until the graph matrix is equally divided.
4. A data mining system for big data analysis according to claim 1, characterized in that the definition of the region parameters of the sampling region is the same as the definition of the region parameters of the sub-region; if the region parameter of the sampling region is negative, it indicates that the size of the sampling region is 0.
5. The data mining system for big data analysis of claim 1, wherein the first hidden layer is calculated as:wherein->And->Tensor matrix representing entity vector and weighted entity vector, respectively,>representing the sum of the diagram matrix and the identity matrix, +.>Representation->Degree matrix of->Weight matrix representing the first hidden layer, +.>Representing the ReLU activation function.
6. The data mining system for big data analysis of claim 1, wherein the first hidden layer is a multi-layer structure, and the calculation formula is as follows:wherein->And->Respectively representing the output and input of the first hidden layer of the l+1 layer, when l=1 +.>Tensor matrix representing entity vectors, when l > 1 +.>Output of the first hidden layer representing the first layer,/->Representing the sum of the diagram matrix and the identity matrix, +.>Representation->Degree matrix of->Weight matrix representing the first hidden layer of the layer l+1,/and>representing the ReLU activation function.
7. The data mining system for big data analysis of claim 1, wherein the first linear layer comprises a plurality of input channels, each input channel comprising a different entity arrangement sequence of graph matrices; the first sampling layer generates a corresponding sampling area on each graph matrix respectively and updates each input graph matrix; the first hidden layer has a channel corresponding to each graph matrix, each channel outputs a group of weighted entity vectors, and the weighted entity vectors of the channels are added and divided by the channel number to be used as the weighted entity vectors of the input full-connection layer.
8. The data mining system for big data analysis of claim 1, wherein the government data identifies that the entity included in the government data includes a government staff post name, a government agency built-in agency name, a government work task name, a policy file name;
and the second sampling layer is arranged in front of the first full-connection layer, extracts an entity from a record of one government work task of a government staff, extracts a weighted entity vector corresponding to the entity extracted from the record of the one government work task from the weighted entity vector to generate a first entity set, performs feature fusion on the first entity set, and then inputs the first full-connection layer, and a classification label output by the first full-connection layer is used for representing offline workload of the one government work task.
9. The data mining system for big data analysis of claim 8, further comprising an online assessment module and a total workload assessment module, wherein the online assessment module collects online work behavior information of a government staff member for a government work task and then generates an online feature vector characterizing the online work behavior;
the on-line work behavior information of one government affair task of a government affair staff comprises the number of pages related to the government affair task, the number of words input on the pages related to the government affair task and the policy file name entity related to the government affair task; the method comprises the steps that online work behavior information of a government work task comprises the number of browsed pages related to the government work task and the number of characters input on the pages related to the government work task, page characteristics and character characteristics are obtained through coding, and entity vectors of policy file name entities related to the page characteristics, the character characteristics and the government work task are spliced to obtain a first spliced vector;
then inputting the first spliced vector into a multi-layer perceptron, wherein the classification label of the multi-layer perceptron is the on-line workload time; extracting the characteristics of an output layer of the input multi-layer perceptron as an on-line characteristic vector;
and the total workload assessment module is used for splicing the characteristic vectors after the characteristic fusion of the on-line characteristic vectors and the first entity set to obtain a second spliced vector, then inputting the second spliced vector into a total workload assessment model, and outputting a grading or grading of the classification label representing the total workload by the total workload assessment model.
10. The data mining system for big data analysis of claim 1, further comprising an portrayal vector module and a vector index module, the portrayal vector module extracting a weighted entity vector of a portrayal object name output by the first hidden layer as the portrayal vector of the portrayal object; the government affair data identifies that the entity contained in the government affair data comprises a photo image, an portrait object name, a character track, a policy file name, a government affair organization name and a government affair personnel post name;
the first full-connection layer outputs two classification labels which are respectively corresponding to the record of illegal behaviors of the portrait object and the record of illegal behaviors of the portrait object;
the vector index module clusters the portrait vectors of all the generated portrait objects, takes the cluster center as a code vector, and maps the portrait vectors in the cluster where the code vector is located with the portrait vectors.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310855939.5A CN116578613B (en) | 2023-07-13 | 2023-07-13 | Data mining system for big data analysis |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310855939.5A CN116578613B (en) | 2023-07-13 | 2023-07-13 | Data mining system for big data analysis |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116578613A CN116578613A (en) | 2023-08-11 |
| CN116578613B true CN116578613B (en) | 2023-09-08 |
Family
ID=87541719
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310855939.5A Active CN116578613B (en) | 2023-07-13 | 2023-07-13 | Data mining system for big data analysis |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116578613B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107798385A (en) * | 2017-12-08 | 2018-03-13 | 电子科技大学 | Recognition with Recurrent Neural Network partially connected method based on block tensor resolution |
| CN111405585A (en) * | 2020-03-19 | 2020-07-10 | 北京联合大学 | Neighbor relation prediction method based on convolutional neural network |
| CN111488734A (en) * | 2020-04-14 | 2020-08-04 | 西安交通大学 | Emotional feature representation learning system and method based on global interaction and syntactic dependency |
| CN112000801A (en) * | 2020-07-09 | 2020-11-27 | 山东师范大学 | Government affair text classification and hot spot problem mining method and system based on machine learning |
| WO2022016561A1 (en) * | 2020-07-22 | 2022-01-27 | 江苏宏创信息科技有限公司 | Ai modeling system and method for policy profiling based on big data |
| CN116226238A (en) * | 2023-05-06 | 2023-06-06 | 合肥尚创信息技术有限公司 | Multi-dimensional heterogeneous big data mining method and system |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170337481A1 (en) * | 2016-05-17 | 2017-11-23 | Xerox Corporation | Complex embeddings for simple link prediction |
| CN113393025A (en) * | 2021-06-07 | 2021-09-14 | 浙江大学 | Non-invasive load decomposition method based on Informer model coding structure |
-
2023
- 2023-07-13 CN CN202310855939.5A patent/CN116578613B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107798385A (en) * | 2017-12-08 | 2018-03-13 | 电子科技大学 | Recognition with Recurrent Neural Network partially connected method based on block tensor resolution |
| CN111405585A (en) * | 2020-03-19 | 2020-07-10 | 北京联合大学 | Neighbor relation prediction method based on convolutional neural network |
| CN111488734A (en) * | 2020-04-14 | 2020-08-04 | 西安交通大学 | Emotional feature representation learning system and method based on global interaction and syntactic dependency |
| CN112000801A (en) * | 2020-07-09 | 2020-11-27 | 山东师范大学 | Government affair text classification and hot spot problem mining method and system based on machine learning |
| WO2022016561A1 (en) * | 2020-07-22 | 2022-01-27 | 江苏宏创信息科技有限公司 | Ai modeling system and method for policy profiling based on big data |
| CN116226238A (en) * | 2023-05-06 | 2023-06-06 | 合肥尚创信息技术有限公司 | Multi-dimensional heterogeneous big data mining method and system |
Non-Patent Citations (1)
| Title |
|---|
| 基于BERT-BLSTM-CRF的政务领域命名实体识别方法;杨春明;魏成志;张晖;赵旭剑;李波;;西南科技大学学报;35(03);86-91 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116578613A (en) | 2023-08-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10599623B2 (en) | Matching multidimensional projections of functional space | |
| US8990268B2 (en) | Domain-specific syntax tagging in a functional information system | |
| CN113822494A (en) | Risk prediction method, device, equipment and storage medium | |
| US20200379958A1 (en) | Dynamic syntactic affinity group formation in a high-dimensional functional information system | |
| Gui et al. | Embedding learning with events in heterogeneous information networks | |
| JP2021504789A (en) | ESG-based corporate evaluation execution device and its operation method | |
| JP2018147351A (en) | Knowledge model construction system and knowledge model construction method | |
| CN112069329A (en) | Text corpus processing method, device, equipment and storage medium | |
| Chopra et al. | Data Science with Python: Combine Python with machine learning principles to discover hidden patterns in raw data | |
| CN116578613B (en) | Data mining system for big data analysis | |
| CN112506930B (en) | Data insight system based on machine learning technology | |
| CN114612246A (en) | Object set identification method and device, computer equipment and storage medium | |
| CN113240325A (en) | Data processing method, device, equipment and storage medium | |
| Sultana et al. | An efficient deep learning method to predict students performance | |
| Ghosh et al. | Understanding machine learning | |
| Mashagba et al. | Using Gradient Boosting Algorithms in Predicting Student Academic Performance | |
| Díaz de la Paz et al. | Weights Estimation in the Completeness Measurement of Bibliographic Metadata | |
| CN117764536B (en) | Innovative entrepreneur project auxiliary management system based on artificial intelligence | |
| Hilmarsson | Using machine learning for predicting the likelihood of upper secondary school student dropout | |
| Mouli et al. | Sentiment analysis to determine employee job satisfaction using machine learning techniques | |
| Shanmugarajah et al. | WoKnack–A Professional Social Media Platform for Women Using Machine Learning Approach | |
| Kumar | ACM Venue Recommendation System | |
| Jahr | Creating an Agglomerative Clustering Approach Using GDELT | |
| Huseyinov¹ et al. | Check for updates | |
| Nilsson | Artificial intelligence application for feature extraction in annual reports: AI-pipeline for feature extraction in Swedish balance sheets from scanned annual reports |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |