CN116702135A - Government affair data monitoring analysis processing method and system based on big data - Google Patents
Government affair data monitoring analysis processing method and system based on big data Download PDFInfo
- Publication number
- CN116702135A CN116702135A CN202310728410.7A CN202310728410A CN116702135A CN 116702135 A CN116702135 A CN 116702135A CN 202310728410 A CN202310728410 A CN 202310728410A CN 116702135 A CN116702135 A CN 116702135A
- Authority
- CN
- China
- Prior art keywords
- data
- government affair
- period
- database
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 83
- 238000004458 analytical method Methods 0.000 title claims abstract description 67
- 238000003672 processing method Methods 0.000 title claims abstract description 8
- 230000005540 biological transmission Effects 0.000 claims abstract description 40
- 238000012545 processing Methods 0.000 claims abstract description 18
- 238000012795 verification Methods 0.000 claims description 4
- 230000006854 communication Effects 0.000 claims description 3
- 238000004891 communication Methods 0.000 claims description 3
- 238000000034 method Methods 0.000 abstract description 11
- 238000004140 cleaning Methods 0.000 abstract description 5
- 238000007726 management method Methods 0.000 description 89
- 238000013523 data management Methods 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007175 bidirectional communication Effects 0.000 description 1
- 238000004883 computer application Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
- G06F21/55—Detecting local intrusion or implementing counter-measures
- G06F21/552—Detecting local intrusion or implementing counter-measures involving long-term monitoring or reporting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/604—Tools and structures for managing or administering access control systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/606—Protecting data by securing the transmission between two devices or processes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Systems or methods specially adapted for specific business sectors, e.g. utilities or tourism
- G06Q50/10—Services
- G06Q50/26—Government or public services
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The application discloses a government affair data monitoring analysis processing method and system based on big data, which solve the technical problem that the government affair data cannot be subjected to data security management and data coverage management in the prior art; after the data architecture management is completed in the database, the real-time supply and demand of corresponding government affair data is safely monitored, the data architecture in the database is safely analyzed and managed, the accuracy of the acquisition of the current data architecture is ensured, the transmission risk of the data architecture in the database is avoided, the high efficiency of government affair data processing is improved, and the government affair data access risk is reduced; and in the process of safety continuous monitoring in the database, the database is subjected to coverage management according to real-time data, so that the data coverage and data cleaning of the administrative data in the database are ensured in time, the storage pressure of the administrative data in the database is reduced, meanwhile, the misoperation of accessing invalid administrative data can be avoided, and the reliability of the administrative data in the database is ensured.
Description
Technical Field
The application relates to the technical field of data monitoring and processing, in particular to a government affair data monitoring, analyzing and processing method and system based on big data.
Background
Government department data gradually presents diversified characteristics under the background of the current big data age, including unstructured, semi-structured and structured different types, so that the electronic government data management level must be improved, the electronic government big data management work is efficiently completed, and the work of collecting, storing, sharing and the like of big data can be effectively realized by carrying out cloud computing-oriented electronic government big data management analysis;
however, in the prior art, the government data cannot analyze and manage the data structure of the government data in the process of collecting and storing, so that the data chain of the government data is incomplete, and meanwhile, the government data cannot be subjected to data security management and data coverage management, so that the government data can not be stored at high cost and can not be used for controlling the security storage of the government data, and the security performance of data transmission is affected;
in view of the above technical drawbacks, a solution is now proposed.
Disclosure of Invention
The application aims to solve the problems, and provides a government affair data monitoring analysis processing method and system based on big data.
The aim of the application can be achieved by the following technical scheme: the utility model provides a government affair data monitoring analysis processing system based on big data, includes the monitoring analysis platform, and monitoring analysis platform communication is connected with:
collecting government affair data of the area according to the monitoring area; collecting government affair data, then summarizing and storing the collected government affair data to construct a database, and monitoring the database through a monitoring analysis platform;
the management and control data architecture management unit is used for carrying out data architecture management on government affair data stored in the database, the data security management unit carries out security analysis management on the data architecture in the database after the management of the government affair data architecture in the database is completed, and the government affair data after the security analysis management is completed carries out coverage management on the government affair data in the database through the data coverage management unit; and after the government affair data is qualified and covered, the data period management unit carries out data period management on the government affair data in the database.
As a preferred embodiment of the present application, the data architecture management unit operates as follows:
marking the collection period of government affair data as a data update period, increasing the data update period in real time along with the system time, and obtaining corresponding data sources according to the government affair data collected in the data update period, wherein the data sources are represented as departments, units and crowd types related to the government affair data, the crowd types can be divided according to occupation and compilation of the masses, binding the data sources with the corresponding government affair data after the data sources are obtained, and dividing the data sources into types according to data flows among the data sources in the data update period, wherein the data sources are divided into a generation end and a receiving end;
dividing the administrative data in the database into i sub-type data, wherein i is a natural number larger than 1, and acquiring a supply and demand range according to data stream transmission of a generation end and a receiving end corresponding to the administrative data, wherein the supply and demand range is represented as a range constructed by data streams formed by a non-unique generation end and a non-unique receiving end corresponding to the administrative data; and acquiring the updating frequency of the transmission data between the generating end corresponding to the same data stream and the receiving end in the supply and demand range and the sum of the overlapping amount of the generating end corresponding to the non-same data stream and the receiving end, and respectively analyzing the updating frequency of the transmission data between the generating end corresponding to the same data stream and the receiving end in the supply and demand range and the sum of the overlapping amount of the generating end corresponding to the non-same data stream and the receiving end.
As a preferred implementation mode of the application, if any numerical value of the update frequency of data transmission between the corresponding generating end and the receiving end of the same data stream and the sum of the overlapping amounts of the corresponding generating end and the receiving end of the non-same data stream in the supply and demand range exceeds a corresponding set threshold, the large supply and demand range of the current government data is judged, the two ends of the real-time data stream in the current government data are recorded, the generating end and the receiving end corresponding to the two ends of the data stream are taken as responsible ends corresponding to the current moment of the data stream, if the data stream transmission is in a data loss state or a disorder state of the receiving end, the current responsible end is taken as a data tracing object, and when a risk occurs, the data stream of the data tracing object is immediately interrupted and the data stream of a risk period is obtained;
if any numerical value of the sum of the update frequency of the transmission data between the corresponding generating end and the receiving end of the same data stream and the overlapping amount of the corresponding generating end and the receiving end of the non-same data stream in the supply and demand range does not exceed the corresponding set threshold, judging that the supply and demand range of the current government data is small, collecting the increase amount of the transmission data of the corresponding generating end and the receiving end of the same data stream in the supply and demand range and the increase amount of the two ends in the same data stream, and if any increase amount numerical value exceeds the corresponding increase amount threshold, carrying out data transmission monitoring on the two ends of the corresponding data stream as responsible ends; if any increment value does not exceed the corresponding increment threshold, the data flow is not monitored at the current moment.
As a preferred embodiment of the present application, the data security management unit operates as follows:
carrying out security management on a data structure corresponding to the current government affair data, acquiring the maximum increment span of the authorized quantity of a corresponding receiving end of the data flow of the same generating end in the data structure corresponding to the government affair data at adjacent time and the overlapping quantity of different ends in the corresponding responsible end in the data flow where the government affair data is located, and comparing the maximum increment span threshold and the overlapping quantity threshold of the two ends respectively: wherein, both ends are denoted as generating end and receiving end.
As a preferred implementation mode of the application, if the maximum increment span of the authorized quantity of the corresponding receiving end of the data flow of the same generating end in the corresponding data framework of the government affair data at adjacent time exceeds a maximum increment span threshold value, or the overlapping quantity of different ends in the corresponding responsible end of the data flow of the government affair data exceeds a two-end overlapping quantity threshold value, judging that the government affair data in the database has safety risk, generating a safety control signal and sending the safety control signal to a monitoring analysis platform;
if the maximum increment span of the authorized amount of the corresponding receiving end of the data flow of the same generating end in the corresponding data framework of the government affair data at adjacent time does not exceed the maximum increment span threshold value and the overlapping amount of different ends in the corresponding responsible end of the data flow of the government affair data does not exceed the two-end overlapping amount threshold value, judging that the security risk of the government affair data in the database is small, generating a continuous monitoring signal and sending the continuous monitoring signal to a monitoring analysis platform.
As a preferred implementation mode of the application, after the monitoring and analyzing platform receives the security control signal, the responsible end of the single data stream of the government affair data is recorded and the data transmission time is recorded, and the identity verification is carried out on the newly added responsible end corresponding to the adjacent transmission time of the responsible end; and after the monitoring analysis platform receives the continuous monitoring signal, performing identity spot check on the data stream of the government affair data corresponding to the newly added responsibility end.
As a preferred embodiment of the present application, the data coverage management unit operates as follows:
acquiring a data updating period of sub-type data corresponding to government data in a database, and dividing the data updating period into a high-frequency updating period and a low-frequency updating period according to the data updating frequency of each period in the data updating period, wherein the dividing mode is a set frequency threshold; acquiring the data updating time of the subtype data corresponding to the government affair data, marking the government affair data generated at the current data updating time as data to be covered, and marking the government affair data before the current data updating time as covered data; and acquiring the current data updating time and the time period of the current data updating period.
As a preferred embodiment of the present application, when the current data update time is in a high-frequency update period, deleting the corresponding covered data and setting the current data to be covered as real-time subtype data after passing through a data stream security operation, wherein the data stream security operation is represented by determining a responsible end and a responsible end identity verification of a data stream corresponding to the subtype data;
and if the access frequency descending speed of the corresponding covered data is continuously reduced and does not tend to be stable, setting the reserved period of the current covered data, deleting the corresponding covered data after the reserved period is passed, setting the current data to be covered as real-time subtype data, and setting the ending time point of the reserved period as the time when the access frequency descending speed is floated and stopped to be stable.
As a preferred embodiment of the present application, the data cycle management unit operates as follows:
acquiring a real-time update buffer time length of a data stream between a generating end and a corresponding receiving end of government affair data in an update period and an access quantity control quantity of the government affair data in the update period in the real-time update buffer time length, and comparing the access quantity control quantity with an update buffer time length threshold and an access quantity control quantity threshold respectively:
if the real-time update buffer time length of the data flow between the generation end and the corresponding receiving end in the update period of the government affair data exceeds the update buffer time length threshold value, managing and controlling the update buffer time length of the corresponding government affair data in the current database; if the access amount control amount of the administrative data currently in the update period in the real-time update buffer time period does not exceed the access amount control amount threshold, generating a control adjustment signal and sending the control adjustment signal to a monitoring analysis platform; if the real-time update buffer time length of the data flow between the generation end and the corresponding receiving end in the update period of the government affair data does not exceed the update buffer time length threshold value and the access quantity control quantity of the current government affair data in the update period in the real-time update buffer time length exceeds the access quantity control quantity threshold value, generating a data period management qualified signal and sending the data period management qualified signal to a monitoring analysis platform.
The government affair data monitoring analysis processing method based on big data comprises the following steps:
s1: data architecture management, wherein after government affair data are collected and stored in a database, the real-time government affair data in the database are subjected to data architecture management; acquiring a data source and a supply and demand range of government affair data according to the acquisition process of the government affair data, constructing a data chain of the government affair data through the data source and the supply and demand range, and carrying out integrity analysis on the data chain;
s2: data safety management, namely after finishing data architecture management in a database, carrying out safety monitoring on real-time supply and demand of corresponding government affair data; the data architecture in the database is subjected to safety analysis management, so that the accuracy of the acquisition of the current data architecture is ensured, the transmission risk of the data architecture in the database is avoided, the high efficiency of government affair data processing is improved, and the government affair data access risk is reduced;
s3: in the process of safety continuous monitoring in the database, the data coverage management is carried out on the database according to real-time data, so that the data coverage and data cleaning of the administrative data in the database are ensured in time, the storage pressure of the administrative data in the database is reduced, meanwhile, the false access behavior of invalid government data can be avoided, and the reliability of the administrative data in the database is ensured;
s4: and (3) carrying out data life cycle management, namely carrying out life cycle management on the administrative data in the database after the administrative data are covered, so as to avoid information island caused by data transmission delay of the corresponding generation end and the receiving end of the administrative data, and be unfavorable for decision making caused by analysis of the administrative data.
Compared with the prior art, the application has the beneficial effects that:
1. in the application, data architecture management is carried out on government affair data stored in a database, a data source and a supply and demand range of the government affair data are acquired according to the acquisition process of the government affair data, a data chain of the government affair data is constructed through the data source and the supply and demand range, and the integrity analysis is carried out on the data chain; the data architecture in the database is subjected to safety analysis management, so that the accuracy of the acquisition of the current data architecture is ensured, the transmission risk of the data architecture in the database is avoided, the high efficiency of government affair data processing is improved, and the access risk of government affair data is reduced.
2. According to the application, the management of the coverage of the administrative data in the database is carried out, so that the timely data coverage and data cleaning of the administrative data in the database are ensured, the storage pressure of the administrative data in the database is reduced, meanwhile, the false access behavior of invalid administrative data can be avoided, and the reliability of the administrative data in the database is ensured; and carrying out data period management on the government affair data in the database, so as to avoid information island caused by data transmission delay of a corresponding generation end and a receiving end of the government affair data, and be unfavorable for decision making deviation caused by analysis of the government affair data.
Drawings
The present application is further described below with reference to the accompanying drawings for the convenience of understanding by those skilled in the art.
FIG. 1 is a schematic block diagram of the present application;
fig. 2 is a flow chart of the method of the present application.
Detailed Description
In order that those skilled in the art will better understand the present application, a technical solution in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments.
Referring to fig. 1, a government affair data monitoring analysis processing system based on big data comprises a monitoring analysis platform, wherein the monitoring analysis platform is in communication connection with a data architecture management unit, a data security management unit, a data coverage manager and a data period management unit, and the monitoring analysis platform is in bidirectional communication connection with the data architecture management unit, the data security management unit, the data coverage manager and the data period management unit;
collecting government affair data of the area according to the monitoring area, wherein the government affair data represent various data resources generated by the government in the process of providing service and managing; collecting government affair data, collecting, storing and constructing a database, and monitoring, managing and controlling the database through a monitoring and analyzing platform;
the monitoring analysis platform generates a data architecture management signal and sends the data architecture management signal to the data architecture management unit, the data architecture management unit carries out data architecture management on government affair data stored in a database after receiving the data architecture management signal, acquires a data source and a supply and demand range of the government affair data according to the collection process of the government affair data, constructs a data chain of the government affair data through the data source and the supply and demand range, and carries out integrity analysis on the data chain;
marking the collection period of government affair data as a data update period, wherein the data update period is increased in real time along with the system time, acquiring corresponding data sources according to the government affair data collected in the data update period, wherein the data sources are represented as departments, units and crowd types related to the government affair data, the crowd types can be divided according to occupation, establishment and the like of the crowd, the data sources are bound with the corresponding government affair data after the data sources are acquired, the data sources are divided into a generation end and a receiving end according to data flows among the data sources in the data update period, particularly when the government affair data are establishment unit wages, the data sources are respectively establishment units and establishment staff, the establishment units are used for carrying out establishment unit wages according to real-time policies and establishment units, the establishment units work as data to the establishment staff, the establishment units are generation ends, the establishment staff are the corresponding generation ends and the receiving ends, and the establishment units are the generated government affair data flows;
dividing the administrative data in the database into i sub-type data, wherein i is a natural number larger than 1, and acquiring a supply and demand range according to data stream transmission of a generation end and a receiving end corresponding to the administrative data, wherein the supply and demand range is represented as a range constructed by data streams formed by a non-unique generation end and a non-unique receiving end corresponding to the administrative data; acquiring the updating frequency of the transmission data between the generating end corresponding to the same data stream and the receiving end in the supply and demand range and the sum of the overlapping amount of the generating end corresponding to the non-same data stream and the receiving end, and respectively analyzing the updating frequency of the transmission data between the generating end corresponding to the same data stream and the receiving end in the supply and demand range and the sum of the overlapping amount of the generating end corresponding to the non-same data stream and the receiving end;
if any numerical value of the sum of the update frequency of data transmitted between the corresponding generating end and the receiving end of the same data stream and the overlapping amount of the corresponding generating end and the receiving end of the non-same data stream in the supply and demand range exceeds a corresponding set threshold, judging that the supply and demand range of the current government data is large, recording the two ends of the real-time data stream in the current government data, taking the generating end and the receiving end corresponding to the two ends of the data stream as the responsible end corresponding to the current moment of the data stream, if the data stream transmission is in data loss or disordered, taking the current responsible end as a data tracing object, immediately interrupting the data stream of the data tracing object when the risk occurs, and acquiring the data stream of a risk period, and controlling the propagation path of the risk data stream to the maximum extent so as to avoid the risk data exceeding the supply and demand range;
if any numerical value of the sum of the update frequency of the transmission data between the corresponding generating end and the receiving end of the same data stream and the overlapping amount of the corresponding generating end and the receiving end of the non-same data stream in the supply and demand range does not exceed the corresponding set threshold, judging that the supply and demand range of the current government data is small, collecting the increase amount of the transmission data of the corresponding generating end and the receiving end of the same data stream in the supply and demand range and the increase amount of the two ends in the same data stream, and if any increase amount numerical value exceeds the corresponding increase amount threshold, carrying out data transmission monitoring on the two ends of the corresponding data stream as responsible ends; if any increment value does not exceed the corresponding increment threshold, not monitoring the data flow at the current moment, monitoring the data flow while ensuring the data transmission safety, controlling the data monitoring cost, and simultaneously ensuring the data tracing efficiency at the abnormal moment;
the supervision and analysis platform generates a data security management signal after finishing management of the administrative data architecture in the database and sends the data security management signal to the data security management unit, and the data security management unit carries out security analysis and management on the data architecture in the database after receiving the data security management signal, so that the accuracy of acquiring the current data architecture is ensured, the transmission risk of the data architecture in the database is avoided, the high efficiency of government data processing is improved, and the government data access risk is reduced;
carrying out security management on a data structure corresponding to current government affair data, obtaining the maximum increment span of the authorized amount of a corresponding adapting end of the data flow corresponding to adjacent time of the same generating end in the data structure corresponding to the government affair data and the overlapping amount of different ends in the corresponding responsible end in the data flow where the government affair data is located, and comparing the maximum increment span of the authorized amount of the corresponding adapting end of the data flow corresponding to adjacent time of the same generating end in the data structure corresponding to the government affair data and the overlapping amount of different ends in the corresponding responsible end in the data flow where the government affair data is located with a maximum increment span threshold value and a two-end overlapping amount threshold value respectively: wherein, two ends are represented as a generating end and a receiving end;
if the maximum increment span of the authorized amount of the corresponding receiving end of the data stream of the same generating end in the corresponding data framework of the government affair data at adjacent time exceeds a maximum increment span threshold value, or the overlapping amount of different two ends in the corresponding responsible end in the data stream of the government affair data exceeds a two-end overlapping amount threshold value, judging that the government affair data in the database has safety risks, generating a safety control signal and sending the safety control signal to a monitoring analysis platform, and after the monitoring analysis platform receives the safety control signal, recording the responsible end of the single data stream of the government affair data and recording the data transmission time, and verifying the identity of the newly-increased responsible end at the adjacent transmission time of the corresponding responsible end;
if the maximum increment span of the authorized amount of the corresponding receiving end of the data stream of the same generating end in the corresponding data framework of the government affair data at adjacent time does not exceed the maximum increment span threshold value and the overlapping amount of different two ends in the corresponding responsible end of the data stream of the government affair data does not exceed the two-end overlapping amount threshold value, judging that the security risk of the government affair data in the database is small, generating a continuous monitoring signal and sending the continuous monitoring signal to a monitoring analysis platform, and after the continuous monitoring signal is received by the monitoring analysis platform, carrying out identity spot check on the data stream of the government affair data corresponding to the newly added responsible end;
the monitoring analysis platform generates a data coverage management signal and sends the data coverage management signal to the data coverage management unit, and the data coverage management unit carries out coverage management on the administrative data in the database after receiving the data coverage management signal, so that the administrative data in the database are timely subjected to data coverage and data cleaning, the storage pressure of the administrative data in the database is reduced, meanwhile, the false access behavior of invalid administrative data can be avoided, and the reliability of the administrative data in the database is ensured;
acquiring a data updating period of sub-type data corresponding to government data in a database, and dividing the data updating period into a high-frequency updating period and a low-frequency updating period according to the data updating frequency of each period in the data updating period, wherein the dividing mode is a set frequency threshold;
acquiring the data updating time of the subtype data corresponding to the government affair data, marking the government affair data generated at the current data updating time as data to be covered, and marking the government affair data before the current data updating time as covered data; acquiring the current data updating time and the period of the current data updating period;
the current data updating time is in a high-frequency updating period, after the data stream security operation, deleting the corresponding covered data and setting the current data to be covered as real-time subtype data, wherein the data stream security operation is represented as determining the responsible end and the identity verification of the responsible end of the data stream corresponding to the subtype data;
monitoring the descending speed of the access frequency of the corresponding covered data after the current data updating time is in a low-frequency updating time period through data stream security operation, if the descending speed of the corresponding access frequency is continuously reduced and does not tend to be stable, setting the reserved time period of the current covered data, deleting the corresponding covered data after the reserved time period is passed, setting the current data to be covered as real-time subtype data, and setting the ending time point of the reserved time period as the time when the descending speed of the access frequency is floated and stopped to be stable; wherein, the data access existing after the covered data falling speed stops floating in the technical proposal is expressed as the access frequency in the data security management process in the database;
the monitoring analysis platform generates a data period management signal and sends the data period management signal to the data period management unit, and the data period management unit carries out data period management on administrative data in the database after receiving the data period management signal, so that information island caused by data transmission delay of a corresponding generation end and a receiving end of the administrative data is avoided, and decision deviation caused by analysis of the administrative data is not facilitated;
acquiring a real-time update buffer time length of a data stream between a generating end and a corresponding receiving end of government affair data in an update period and an access quantity control quantity of the current government affair data in the update period in the real-time update buffer time length, and comparing the real-time update buffer time length of the data stream between the generating end and the corresponding receiving end of government affair data in the update period and the access quantity control quantity of the current government affair data in the update period in the real-time update buffer time length with an update buffer time length threshold and an access quantity control quantity threshold respectively:
if the real-time update buffer time length of the data flow between the generating end and the corresponding receiving end in the update period of the government affair data exceeds the update buffer time length threshold value, judging that the generating end and the receiving end of the corresponding government affair data in the current database have information island risks, and managing and controlling the update buffer time length of the corresponding government affair data in the current database; the information island refers to a computer application system which is not mutually associated and assisted in function, information is not shared and exchanged and information, business flow and application are mutually disjointed, and in the application, the information island is expressed as that the transmission efficiency of data flow after data in a generating end and a receiving end is updated cannot meet the requirement;
if the access quantity control quantity of the administrative data currently in the update period in the real-time update buffer time does not exceed the access quantity control quantity threshold value, judging that the influence of information island corresponding to the administrative data in the current database is large, generating a management and control adjustment signal and sending the management and control adjustment signal to a monitoring analysis platform, and controlling the access quantity of the administrative data in the database in the update buffer time after the monitoring analysis platform receives the management and control adjustment signal;
if the real-time update buffer time length of the data flow between the generation end and the corresponding receiving end in the update period of the government affair data does not exceed the update buffer time length threshold value and the access quantity control quantity of the current government affair data in the update period in the real-time update buffer time length exceeds the access quantity control quantity threshold value, judging that the data period management of the government affair data in the database is qualified, generating a data period management qualified signal and sending the data period management qualified signal to a monitoring analysis platform;
referring to fig. 2, a government affair data monitoring analysis processing method based on big data includes the following steps:
s1: data architecture management, wherein after government affair data are collected and stored in a database, the real-time government affair data in the database are subjected to data architecture management; acquiring a data source and a supply and demand range of government affair data according to the acquisition process of the government affair data, constructing a data chain of the government affair data through the data source and the supply and demand range, and carrying out integrity analysis on the data chain;
s2: data safety management, namely after finishing data architecture management in a database, carrying out safety monitoring on real-time supply and demand of corresponding government affair data; the data architecture in the database is subjected to safety analysis management, so that the accuracy of the acquisition of the current data architecture is ensured, the transmission risk of the data architecture in the database is avoided, the high efficiency of government affair data processing is improved, and the government affair data access risk is reduced;
s3: in the process of safety continuous monitoring in the database, the data coverage management is carried out on the database according to real-time data, so that the data coverage and data cleaning of the administrative data in the database are ensured in time, the storage pressure of the administrative data in the database is reduced, meanwhile, the false access behavior of invalid government data can be avoided, and the reliability of the administrative data in the database is ensured;
s4: and (3) carrying out data life cycle management, namely carrying out life cycle management on the administrative data in the database after the administrative data are covered, so as to avoid information island caused by data transmission delay of the corresponding generation end and the receiving end of the administrative data, and be unfavorable for decision making caused by analysis of the administrative data.
When the application is used, the government affair data acquisition of the area is carried out according to the monitoring area; collecting government affair data, then summarizing and storing the collected government affair data to construct a database, and monitoring the database through a monitoring analysis platform; the management and control data architecture management unit carries out data architecture management on government affair data stored in the database, the supervision and analysis platform carries out security analysis management on the data architecture in the database after finishing management on the government affair data architecture in the database, and carries out coverage management on the government affair data in the database through the data coverage management unit on the government affair data which finish the security analysis management; and after the government affair data is qualified and covered, the data period management unit carries out data period management on the government affair data in the database.
The preferred embodiments of the application disclosed above are intended only to assist in the explanation of the application. The preferred embodiments are not intended to be exhaustive or to limit the application to the precise form disclosed. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the application and the practical application, to thereby enable others skilled in the art to best understand and utilize the application. The application is limited only by the claims and the full scope and equivalents thereof.
Claims (10)
1. The government affair data monitoring analysis processing system based on big data is characterized by comprising a monitoring analysis platform, wherein the monitoring analysis platform is in communication connection with:
collecting government affair data of the area according to the monitoring area; collecting government affair data, then summarizing and storing the collected government affair data to construct a database, and monitoring the database through a monitoring analysis platform;
the management and control data architecture management unit is used for carrying out data architecture management on government affair data stored in the database, the data security management unit carries out security analysis management on the data architecture in the database after the management of the government affair data architecture in the database is completed, and the government affair data after the security analysis management is completed carries out coverage management on the government affair data in the database through the data coverage management unit; and after the government affair data is qualified and covered, the data period management unit carries out data period management on the government affair data in the database.
2. The government affair data monitoring, analyzing and processing system based on big data according to claim 1, wherein the data architecture management unit operates as follows:
marking the collection period of government affair data as a data update period, increasing the data update period in real time along with the system time, and obtaining corresponding data sources according to the government affair data collected in the data update period, wherein the data sources are represented as departments, units and crowd types related to the government affair data, the crowd types can be divided according to occupation and compilation of the masses, binding the data sources with the corresponding government affair data after the data sources are obtained, and dividing the data sources into types according to data flows among the data sources in the data update period, wherein the data sources are divided into a generation end and a receiving end;
dividing the administrative data in the database into i sub-type data, wherein i is a natural number larger than 1, and acquiring a supply and demand range according to data stream transmission of a generation end and a receiving end corresponding to the administrative data, wherein the supply and demand range is represented as a range constructed by data streams formed by a non-unique generation end and a non-unique receiving end corresponding to the administrative data; and acquiring the updating frequency of the transmission data between the generating end corresponding to the same data stream and the receiving end in the supply and demand range and the sum of the overlapping amount of the generating end corresponding to the non-same data stream and the receiving end, and respectively analyzing the updating frequency of the transmission data between the generating end corresponding to the same data stream and the receiving end in the supply and demand range and the sum of the overlapping amount of the generating end corresponding to the non-same data stream and the receiving end.
3. The government affair data monitoring analysis processing system based on big data according to claim 2, wherein if any value of the sum of the update frequency of data transmitted between the generating end and the receiving end corresponding to the same data stream in the supply and demand range and the overlapping amount of the generating end and the receiving end corresponding to the non-same data stream exceeds a corresponding set threshold, the supply and demand range of the current government affair data is judged to be big, the two ends of the real-time data stream in the current government affair data are recorded, the generating end and the receiving end corresponding to the two ends of the data stream are taken as responsible ends corresponding to the current moment of the data stream, if the data stream transmission has data loss or disorder of the receiving end, the current responsible end is taken as a data tracing object, and when a risk occurs, the data stream of the data tracing object is interrupted and the data stream of a risk period is obtained;
if any numerical value of the sum of the update frequency of the transmission data between the corresponding generating end and the receiving end of the same data stream and the overlapping amount of the corresponding generating end and the receiving end of the non-same data stream in the supply and demand range does not exceed the corresponding set threshold, judging that the supply and demand range of the current government data is small, collecting the increase amount of the transmission data of the corresponding generating end and the receiving end of the same data stream in the supply and demand range and the increase amount of the two ends in the same data stream, and if any increase amount numerical value exceeds the corresponding increase amount threshold, carrying out data transmission monitoring on the two ends of the corresponding data stream as responsible ends; if any increment value does not exceed the corresponding increment threshold, the data flow is not monitored at the current moment.
4. The government affair data monitoring, analyzing and processing system based on big data according to claim 1, wherein the data safety management unit operates as follows:
carrying out security management on a data structure corresponding to the current government affair data, acquiring the maximum increment span of the authorized quantity of a corresponding receiving end of the data flow of the same generating end in the data structure corresponding to the government affair data at adjacent time and the overlapping quantity of different ends in the corresponding responsible end in the data flow where the government affair data is located, and comparing the maximum increment span threshold and the overlapping quantity threshold of the two ends respectively: wherein, both ends are denoted as generating end and receiving end.
5. The big data-based government affair data monitoring analysis processing system according to claim 4, wherein if the maximum increment span of the authorized amount of the corresponding receiving end of the data flow of the same generating end in the adjacent moment in the government affair data corresponding data structure exceeds a maximum increment span threshold value, or the overlapping amount of different ends in the corresponding responsible end in the data flow of the government affair data exceeds a two-end overlapping amount threshold value, judging that the security risk exists in the government affair data in the database, generating a security control signal and sending the security control signal to a monitoring analysis platform;
if the maximum increment span of the authorized amount of the corresponding receiving end of the data flow of the same generating end in the corresponding data framework of the government affair data at adjacent time does not exceed the maximum increment span threshold value and the overlapping amount of different ends in the corresponding responsible end of the data flow of the government affair data does not exceed the two-end overlapping amount threshold value, judging that the security risk of the government affair data in the database is small, generating a continuous monitoring signal and sending the continuous monitoring signal to a monitoring analysis platform.
6. The government affair data monitoring analysis processing system based on big data according to claim 5, wherein after the monitoring analysis platform receives the security control signal, the responsibility end of the single data stream of the government affair data is recorded and the data transmission time is recorded, and identity verification is performed on the newly added responsibility end adjacent to the transmission time of the corresponding responsibility end; and after the monitoring analysis platform receives the continuous monitoring signal, performing identity spot check on the data stream of the government affair data corresponding to the newly added responsibility end.
7. The government affair data monitoring, analyzing and processing system based on big data according to claim 1, wherein the data coverage management unit operates as follows:
acquiring a data updating period of sub-type data corresponding to government data in a database, and dividing the data updating period into a high-frequency updating period and a low-frequency updating period according to the data updating frequency of each period in the data updating period, wherein the dividing mode is a set frequency threshold; acquiring the data updating time of the subtype data corresponding to the government affair data, marking the government affair data generated at the current data updating time as data to be covered, and marking the government affair data before the current data updating time as covered data; and acquiring the current data updating time and the time period of the current data updating period.
8. The big data-based government affair data monitoring analysis processing system according to claim 7, wherein the current data update time is in a high-frequency update period, after passing through a data stream security operation, deleting the corresponding covered data and setting the current data to be covered as real-time subtype data, wherein the data stream security operation is represented by determining a responsible end of a data stream corresponding to subtype data and verifying the identity of the responsible end; and if the access frequency descending speed of the corresponding covered data is continuously reduced and does not tend to be stable, setting the reserved period of the current covered data, deleting the corresponding covered data after the reserved period is passed, setting the current data to be covered as real-time subtype data, and setting the ending time point of the reserved period as the time when the access frequency descending speed is floated and stopped to be stable.
9. The government affair data monitoring, analyzing and processing system based on big data according to claim 1, wherein the data period management unit operates as follows:
acquiring a real-time update buffer time length of a data stream between a generating end and a corresponding receiving end of government affair data in an update period and an access quantity control quantity of the government affair data in the update period in the real-time update buffer time length, and comparing the access quantity control quantity with an update buffer time length threshold and an access quantity control quantity threshold respectively:
if the real-time update buffer time length of the data flow between the generation end and the corresponding receiving end in the update period of the government affair data exceeds the update buffer time length threshold value, managing and controlling the update buffer time length of the corresponding government affair data in the current database; if the access amount control amount of the administrative data currently in the update period in the real-time update buffer time period does not exceed the access amount control amount threshold, generating a control adjustment signal and sending the control adjustment signal to a monitoring analysis platform; if the real-time update buffer time length of the data flow between the generation end and the corresponding receiving end in the update period of the government affair data does not exceed the update buffer time length threshold value and the access quantity control quantity of the current government affair data in the update period in the real-time update buffer time length exceeds the access quantity control quantity threshold value, generating a data period management qualified signal and sending the data period management qualified signal to a monitoring analysis platform.
10. The government affair data monitoring analysis processing method based on big data is characterized by comprising the government affair data monitoring analysis processing system based on big data according to any one of claims 1-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310728410.7A CN116702135A (en) | 2023-06-19 | 2023-06-19 | Government affair data monitoring analysis processing method and system based on big data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310728410.7A CN116702135A (en) | 2023-06-19 | 2023-06-19 | Government affair data monitoring analysis processing method and system based on big data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116702135A true CN116702135A (en) | 2023-09-05 |
Family
ID=87823628
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310728410.7A Pending CN116702135A (en) | 2023-06-19 | 2023-06-19 | Government affair data monitoring analysis processing method and system based on big data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116702135A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117251499A (en) * | 2023-11-15 | 2023-12-19 | 山东光合云谷大数据有限公司 | Data acquisition system |
-
2023
- 2023-06-19 CN CN202310728410.7A patent/CN116702135A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117251499A (en) * | 2023-11-15 | 2023-12-19 | 山东光合云谷大数据有限公司 | Data acquisition system |
| CN117251499B (en) * | 2023-11-15 | 2024-02-06 | 山东光合云谷大数据有限公司 | Data acquisition system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102081622B (en) | Method and device for evaluating system health degree | |
| CN105631026A (en) | Security data analysis system | |
| CN107273273A (en) | A kind of distributed type assemblies hardware fault method for early warning and system | |
| CN106844161A (en) | Abnormal monitoring and Forecasting Methodology and system in a kind of carrier state stream calculation system | |
| CN116702135A (en) | Government affair data monitoring analysis processing method and system based on big data | |
| CN102982678A (en) | Traffic data information service system and method for realizing traffic data information service | |
| CN104376365A (en) | Method for constructing information system running rule libraries on basis of association rule mining | |
| US11704220B2 (en) | Machine learning based data monitoring | |
| CN109460339A (en) | The streaming computing system of log | |
| CN107748946A (en) | Electric power optical transmission device state-detection evaluation system | |
| CN115733762A (en) | Monitoring system with big data analysis capability | |
| CN109493218A (en) | A kind of marketing message update system and method based on block chain | |
| CN111222968A (en) | Enterprise tax risk management and control method and system | |
| CN116863723B (en) | Use method of digital twin base | |
| CN106649034A (en) | Visual intelligent operation and maintenance method and platform | |
| JP2022533803A (en) | System and method for secure defense against collusion attacks in the Internet of vehicles | |
| CN111221802A (en) | Digital asset risk management and control system and method based on big data | |
| CN106059065B (en) | Based on the adaptive emergent control decision-making technique of safety and stability in value control measure | |
| CN106651145A (en) | Spare part management system and method | |
| Lou et al. | Alarm compression based on machine learning and association rules mining in optical networks | |
| Pan et al. | Secure Data Access and Consensus Algorithm based on Consortium Blockchain in LVC | |
| CN114691633A (en) | Log management method, device and storage medium for rail transit signal system | |
| CN106230615A (en) | A kind of business work order activating method based on intelligence route searching | |
| CN113159941A (en) | Intelligent streaming transaction execution method and device | |
| CN110971645B (en) | Industrial intelligent energy-saving remote service system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |