CN112035438A - Government affair big data platform system - Google Patents
Government affair big data platform system Download PDFInfo
- Publication number
- CN112035438A CN112035438A CN202010903094.9A CN202010903094A CN112035438A CN 112035438 A CN112035438 A CN 112035438A CN 202010903094 A CN202010903094 A CN 202010903094A CN 112035438 A CN112035438 A CN 112035438A
- Authority
- CN
- China
- Prior art keywords
- data
- subsystem
- service
- resource
- department
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3055—Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/285—Clustering or classification
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Quality & Reliability (AREA)
- Computing Systems (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The application relates to a big data platform system of government affairs belongs to computer technology field, and this system includes: the data convergence subsystem converges the heterogeneous multi-source data; the data asset subsystem carries out full-life cycle management on the data in the data aggregation subsystem according to government affair data asset management rules; the data service subsystem determines the use mode of the data in the data aggregation subsystem; the data management subsystem manages and controls the data quality in the data aggregation subsystem; monitoring service operation and platform operation and maintenance in the system; the problems that the data exchanged by the existing data exchange platform is low in quality, the standards are not uniform, and the sensitive data has safety risks can be solved; through the collection, exchange and management of heterogeneous data, the efficiency and accuracy of data exchange can be improved.
Description
Technical Field
The application relates to a government affair big data platform system, and belongs to the technical field of computers.
Background
The advent of the big data era brought new opportunities for government regulatory changes. In order to break through various information barriers and data islands, some local governments build data exchange platforms, and data exchange is realized in a data interface mode.
However, the data types exchanged by the existing data exchange platform are limited to formatted data, and for unformatted data (such as photos, audios and videos, and the like), the demand department and the supply department are still required to transmit data to each other in a point-to-point manner, and cannot be shared and supervised through tools. With the increasing of government affair data, the magnitude and complexity of the data are continuously improved, and the traditional data exchange platform is difficult to process ultra-large-scale data, so that the sharing efficiency of the government affair data is limited.
Disclosure of Invention
The application provides a big data platform system of government affairs, can solve the problem that the data quality that current data exchange platform exchanged is not high, the standard is not unified, there is safe risk in sensitive data etc.. The application provides the following technical scheme: there is provided a government affairs big data platform system, the system comprising:
the data convergence subsystem is used for converging heterogeneous multi-source data;
the data asset subsystem is used for carrying out full life cycle management on the data in the data aggregation subsystem according to government affair data asset management rules;
the data service subsystem is used for determining the use mode of the data in the data aggregation subsystem;
the data management subsystem is used for managing and controlling the data quality in the data aggregation subsystem; and monitoring service operation and platform operation and maintenance in the system.
Optionally, the data access manner of the data aggregation subsystem includes at least one of the following manners:
the database mode supports various main stream databases of MySQL, Oracle, SqlServer, Hive and/or HBase;
the file mode supports formatted files and unformatted files; through data reporting, the extraction of data in the formatted file is supported and the data is stored in a relational database; uploading and gathering the unformatted files through the files;
and the interface mode supports RESTful and Web Service interface types.
Optionally, the data aggregation subsystem is configured to:
the operation is periodically operated according to a scheduling plan to realize data aggregation;
monitoring a running process of a job using job running monitoring, the running process including at least one of: run time, run status, successfully processed data, query data, run logs, and historical run conditions;
monitoring hardware information of a server running the data aggregation subsystem using device operation monitoring, the hardware information including at least one of: CPU occupation, memory occupation and disk occupation;
and when the hardware information indicates that the server has abnormity, early warning is carried out.
Optionally, the data asset subsystem is configured to:
classifying the data in the data aggregation subsystem according to a preset resource cataloging rule to obtain a resource classification;
registering resource information corresponding to the data in the data aggregation subsystem, and establishing a data association table corresponding to the registered resource information, wherein the data association table is used for associating the resource information with the data in the data aggregation subsystem; the resource information comprises basic information and resource information items; the basic information comprises a resource name, a resource classification, a resource abstract, a resource type, a sharing mode and/or an updating period; the resource information item is used for describing the classified resource;
issuing the registered resource information and providing inquiry service of the issued resource information;
when a viewing request of target data corresponding to target resource information is received, if the sharing type of the target data is conditional sharing, a demand department sends a resource viewing demand to a resource providing department so that the resource providing department can review the resource viewing demand; the resource viewing demand comprises a resource information item, a demand reason and demand time of the target resource information;
after the resource providing department passes the examination and verification of the resource viewing requirement, recording in a data management department to form a three-party protocol; after the examination and the record are passed, allowing a demand department to use target data corresponding to the target resource information; and the demand department calls data according to the sharing mode corresponding to the target data.
Optionally, the data service subsystem is configured to:
receiving an interface service registered and issued by an interface providing department, wherein the interface service is used for calling external data or feeding back a result according to an input condition, and the interface service comprises at least one of the following information: basic information, out-participation-parameter information, example codes and error codes;
and generating a service description document according to the interface service, and providing an export function of the service description document so as to enable a developer to refer to and call.
Optionally, the data service subsystem is further configured to:
receiving a demand application provided by a service demand department;
sending the requirement application to an application auditing department for auditing;
after the application auditing department passes the auditing, distributing an authorized password for the service demand department;
receiving a document calling request sent by a service demand department by using a corresponding authorization password;
providing the service description document for the service demand department based on the authorized password, and recording calling information of the service description document, wherein the calling information comprises at least one of the following information: calling a service name, calling an IP, calling time, calling parameters, and calling status.
Optionally, the data service subsystem is further configured to:
determining whether the service demand department is reliable or not according to a preset white list;
determining whether the calling frequency of the service description document requested by the service demand department exceeds a preset threshold value;
and when the service demand department is reliable and the calling frequency does not exceed the preset threshold value, executing the steps of providing the service description document for the service demand department based on the authorized password and recording the calling information of the service description document.
Optionally, the data governance subsystem is configured to:
setting governing standards, wherein the governing standards comprise management standards for data desensitization, data authority, data code values and data verification;
making an ETL operation and a scheduling plan of the ETL operation according to the treatment standard;
scheduling the ETL operation according to the scheduling plan so as to clean data according to the treatment standard to obtain a treatment result;
and processing the treatment result.
Optionally, the data governance subsystem is configured to:
classifying data which does not meet the governing standard into question data;
displaying the data field content and the question reason of the question data;
checking a source business database of the query data;
adjusting the related business operation of the source business database according to the checking result;
data is extracted through the ETL job.
The beneficial effect of this application lies in: converging heterogeneous multi-source data through a data converging subsystem; the data asset subsystem carries out full-life cycle management on the data in the data aggregation subsystem according to government affair data asset management rules; the data service subsystem determines the use mode of the data in the data aggregation subsystem; the data management subsystem manages and controls the data quality in the data aggregation subsystem; monitoring service operation and platform operation and maintenance in the system; the problems that the data exchanged by the existing data exchange platform is low in quality, the standards are not uniform, and the sensitive data has safety risks can be solved; through the collection, exchange and management of heterogeneous data, the efficiency and accuracy of data exchange can be improved.
In addition, the data aggregation subsystem and the data asset subsystem eliminate the problems of 'shaft type' service, 'data isolated island', repeated construction, resource waste and the like in the construction of the traditional information platform. All business systems are built by all departments, the system integration level is low, data information is scattered, and the data standards are not uniform. The method is characterized in that four basic libraries including a population basic information resource library, a legal basic information resource library, a geographic space information resource library, an electronic certificate information resource library and the like are integrated through a government affair big data platform, and subject libraries including industrial economy, safety and the like are integrated, so that basic data resources are provided for various applications of the platform, and resource integration and utilization rate are improved.
In addition, the data sharing exchange platform can realize multi-user access and multi-application support; by integrating data sharing channels among users, a platform support is provided for safe, efficient, ordered and reliable data sharing and exchange. Through the unified integration of platform resources, exchange modes such as data available invisibility, data service calling and the like are adopted in a data storage and exchange mechanism, and the exchange efficiency is greatly improved.
In addition, as IT informatization and government business are combined closely, business requirements have the characteristics of short period, different requirements and simple requirements. The traditional construction mode has the disadvantages of multiple purchasing processes, long deployment time and incapability of quickly adapting to service demand changes. The government affair big data platform can support data interaction between a data service system and service systems of other relevant departments, so that the problems of reliability, interaction, safety and the like of various data layers are omitted when the service system is deployed and on-line, and the government informatization efficiency is greatly improved.
In addition, by enhancing the acquisition, organization, analysis and decision of government affair data, the government affair information resources are uniformly managed, developed and utilized according to laws and regulations and the requirements of each department, the utilization rate of the data resources can be improved, repeated construction is avoided, and the maintenance cost is reduced; and the decision-making efficiency is further improved by deeply mining government affair information resources.
The foregoing description is only an overview of the technical solutions of the present application, and in order to make the technical solutions of the present application more clear and clear, and to implement the technical solutions according to the content of the description, the following detailed description is made with reference to the preferred embodiments of the present application and the accompanying drawings.
Drawings
FIG. 1 is a schematic structural diagram of a government affairs big data platform system according to an embodiment of the present application;
FIG. 2 is a schematic flow diagram of a data asset subsystem provided by an embodiment of the present application;
fig. 3 is a flow diagram of a data service subsystem according to an embodiment of the present application.
Detailed Description
The following detailed description of embodiments of the present application will be described in conjunction with the accompanying drawings and examples. The following examples are intended to illustrate the present application but are not intended to limit the scope of the present application.
Fig. 1 is a schematic structural diagram of a government affairs big data platform system according to an embodiment of the present application, and as shown in fig. 1, the system at least includes:
and the data aggregation subsystem 110 is used for aggregating data of heterogeneous multiple sources. Heterogeneous data includes structured data (e.g., Oracle, Mysql, SQLServer), semi-structured data (e.g., XML, JSON), and unstructured data (e.g., text, pictures, images, audio-video).
A data asset subsystem 120, configured to perform full-life cycle management on the data in the data aggregation subsystem 110 according to government affair data asset management rules; and the data service subsystem 130 is used for determining the use mode of the data in the data aggregation subsystem. Through the data asset subsystem 120 and the data service subsystem 130, sharing and exchange of different types of data can be achieved, corresponding service interfaces are automatically generated when the resource directories are issued, manual development of technicians is not needed, and efficiency and accuracy are improved.
The data governance subsystem 140 is used for managing and controlling the data quality in the data aggregation subsystem; and monitoring service operation and platform operation and maintenance in the system. The data management subsystem 140 formulates standard rules such as metadata standard, data desensitization rule, data authority and the like, so that data auditing, desensitization and conversion are realized, and data quality and privacy safety are ensured.
The data asset subsystem 120 faces government office staff, and achieves sharing exchange of independent management and control data of the government office staff through functions of requirement proposing, requirement checking, use monitoring and the like, and sharing efficiency is improved.
In this embodiment, the government affair big data platform adopts a Hadoop big data architecture system, and supports a high-concurrency application scenario of hundreds of millions of data-second-level processing.
Optionally, the data access manner of the data aggregation subsystem 110 includes at least one of the following manners: the database mode supports various main stream databases of MySQL, Oracle, SqlServer, Hive and/or HBase; the file mode supports formatted files and unformatted files; through data reporting, the extraction of data in the formatted file is supported and the data is stored in a relational database; uploading and gathering the unformatted files through the files; and the interface mode is realized by a key job, and the RESTful and Web Service interface types are supported. The data aggregation subsystem 110 realizes structured, semi-structured, and unstructured heterogeneous data aggregation through various acquisition modes such as database operation, interface operation, file uploading, network capturing, and the like, and breaks through data walls.
The data aggregation subsystem 110 periodically runs the job according to the scheduling plan to realize data aggregation; monitoring a running process of a job using job running monitoring, the running process including at least one of: run time, run status, successfully processed data, query data, run logs, and historical run conditions; monitoring hardware information of a server running the data aggregation subsystem using device operation monitoring, the hardware information including at least one of: CPU occupation, memory occupation and disk occupation; and when the hardware information indicates that the server has abnormity, early warning is carried out.
The data asset subsystem 120 implements full life cycle management from resource cataloging, resource registration, data association, resource release, data service generation, data sharing requirement proposing, auditing and calling according to government affair data assets management thinking.
Referring to FIG. 2, the data asset subsystem 120 is configured to: classifying the data in the data aggregation subsystem according to a preset resource cataloging rule to obtain resource classification (namely resource cataloging); registering (namely resource registration) resource information corresponding to the data in the data aggregation subsystem, and establishing a data association table corresponding to the registered resource information, wherein the data association table is used for associating the resource information with the data in the data aggregation subsystem; the resource information comprises basic information and resource information items; the basic information comprises a resource name, a resource classification, a resource abstract, a resource type, a sharing mode and/or an updating period; the resource information item is used for describing the classified resource; issuing the registered resource information (namely, issuing the resource), and providing query service of the issued resource information; when a viewing request (namely, a resource requirement) for target data corresponding to target resource information is received, if the sharing type of the target data is conditional sharing, a requirement department sends the resource viewing requirement to a resource providing department so that the resource providing department can review the resource viewing requirement (namely, requirement review); the resource viewing demand comprises a resource information item, a demand reason and demand time of the target resource information; after the resource providing department passes the examination and verification of the resource viewing requirement, recording in a data management department to form a three-party protocol; after the examination and the record pass, allowing a demand department to use target data (namely resource use) corresponding to the target resource information; and the demand department calls data (namely, uses monitoring) according to the sharing mode corresponding to the target data.
Optionally, the sharing type of the data includes: unconditional sharing, conditional sharing and no sharing. When the sharing type of the data is unconditional sharing, the demand department can directly check the data; when the sharing type of the data is not shared, the demand department cannot view the data.
The sharing mode of the data comprises online browsing, file downloading and data service. When the sharing mode is online browsing, the demand department can only browse data on the page and cannot operate the data; when the sharing mode is file downloading, the demand department can export the data into a table; when the sharing mode is data service, the demand department can call data through a service interface automatically generated by resources for the direct use of a business system.
The data service subsystem 130 mainly solves the problem of data utilization, a demand department calls data through interface service, and the platform authorizes, authenticates and monitors the use of the data to ensure the safety of data sharing.
The data service subsystem 130 provides third-party service registration and shared use, and there are two main service scenarios, one is feedback of conditional data or results. For example, in the administrative examination and approval process, in order to reduce the repeated submission of the relevant license by the office, the personal information interface can be called according to the name and the identity card information of the office to obtain the license information of the office. Such an interface that needs to return specific information or results of a certain type according to input conditions can be registered and published in the data service subsystem. Yet another business scenario is the invocation of external data. The government data, besides the data produced by the service line, also includes a part of data collected from outside, such as data of network operators, meteorological data, data of related monitoring laboratories, etc., and the sharing of these data is generally provided by the providing unit through interfaces, which can be registered and released in the data service subsystem in order to facilitate uniform sharing management.
Referring to fig. 3 (black background frame for front-end interface operation, white background frame for back-end server operation), the data service subsystem 130 is configured to: receiving interface services (namely foreground registration and rear-end release) registered and released by an interface providing department, wherein the interface services are used for calling external data or feeding back results according to input conditions, and comprise at least one of the following information: basic information, out-participation-parameter information, example codes and error codes; and generating a service description document according to the interface service, and providing an export function of the service description document so as to enable a developer to refer to and call. Specifically, generating a service description document according to the interface service and providing a function of exporting the service description document includes: receiving a demand application (namely a front-end application and a back-end supply demand) provided by a service demand department; sending the requirement application to an application auditing department for auditing; after the application auditing department passes the auditing, distributing an authorization password (namely front-end auditing and back-end authorization) for the service demand department; receiving a document calling request sent by a service demand department by using a corresponding authorization password; providing the service description document (namely front-end calling and back-end authentication) for the service demand department based on the authorized password, and recording calling information (namely front-end analysis and back-end monitoring) of the service description document, wherein the calling information comprises at least one of the following information: calling a service name, calling an IP, calling time, calling parameters, and calling status.
Optionally, the data service subsystem 130 further provides a white list function, so as to ensure the safety and reliability of the service caller; and meanwhile, a frequency control function is provided, so that the phenomenon that the service is simultaneously called and the network is blocked is prevented. In this case, the data service subsystem 130 is further configured to: determining whether the service demand department is reliable or not according to a preset white list; determining whether the calling frequency of the service description document requested by the service demand department exceeds a preset threshold value; and when the service demand department is reliable and the calling frequency does not exceed the preset threshold value, executing the steps of providing the service description document for the service demand department based on the authorized password and recording the calling information of the service description document.
On one hand, the data governance subsystem 140 ensures the data quality through the creation and execution of data standards; on the other hand, the safety and stability of data in the transmission and storage processes are guaranteed by monitoring service operation, platform operation and maintenance and the like.
A data governance subsystem 140 for: setting governing standards, wherein the governing standards comprise management standards for data desensitization, data authority, data code values and data verification; making an ETL operation and a scheduling plan of the ETL operation according to the treatment standard; scheduling the ETL operation according to the scheduling plan so as to clean data according to the treatment standard to obtain a treatment result; and processing the treatment result.
Wherein, the treatment result is processed, which comprises the following steps: classifying data which does not meet the governing standard into question data; displaying the data field content and the question reason of the question data; checking a source business database of the query data; adjusting the related business operation of the source business database according to the checking result; data is extracted through the ETL job.
Data desensitization is mainly set for desensitization of sensitive data such as names, mobile phone numbers, identity card numbers, addresses and the like. Data authority deals with the problem of checking one or more sources. One source refers to that each piece of data corresponds to one source business database; the multi-source check refers to correction and audit of data from multiple sources. The data code value is a setting for a standard data code such as a gender code, a provincial code, a academic code, a professional code, and the like. Data checks are settings of the conversion rules, such as non-null settings, fixed length settings, etc.
The display mode can be quickly and flexibly customized, such as: the BI data chart is flexibly defined, and various visualization charts such as lists, line graphs, bar charts, pie charts, funnel graphs and radar charts are supported.
In summary, the government affair big data platform method provided by the embodiment converges heterogeneous multi-source data through the data convergence subsystem; the data asset subsystem carries out full-life cycle management on the data in the data aggregation subsystem according to government affair data asset management rules; the data service subsystem determines the use mode of the data in the data aggregation subsystem; the data management subsystem manages and controls the data quality in the data aggregation subsystem; monitoring service operation and platform operation and maintenance in the system; the problems that the data exchanged by the existing data exchange platform is low in quality, the standards are not uniform, and the sensitive data has safety risks can be solved; through the collection, exchange and management of heterogeneous data, the efficiency and accuracy of data exchange can be improved.
In addition, the data aggregation subsystem and the data asset subsystem eliminate the problems of 'shaft type' service, 'data isolated island', repeated construction, resource waste and the like in the construction of the traditional information platform. All business systems are built by all departments, the system integration level is low, data information is scattered, and the data standards are not uniform. The method is characterized in that four basic libraries including a population basic information resource library, a legal basic information resource library, a geographic space information resource library, an electronic certificate information resource library and the like are integrated through a government affair big data platform, and subject libraries including industrial economy, safety and the like are integrated, so that basic data resources are provided for various applications of the platform, and resource integration and utilization rate are improved.
In addition, the data sharing exchange platform can realize multi-user access and multi-application support; by integrating data sharing channels among users, a platform support is provided for safe, efficient, ordered and reliable data sharing and exchange. Through the unified integration of platform resources, exchange modes such as data available invisibility, data service calling and the like are adopted in a data storage and exchange mechanism, and the exchange efficiency is greatly improved.
In addition, as IT informatization and government business are combined closely, business requirements have the characteristics of short period, different requirements and simple requirements. The traditional construction mode has the disadvantages of multiple purchasing processes, long deployment time and incapability of quickly adapting to service demand changes. The government affair big data platform can support data interaction between a data service system and service systems of other relevant departments, so that the problems of reliability, interaction, safety and the like of various data layers are omitted when the service system is deployed and on-line, and the government informatization efficiency is greatly improved.
In addition, by enhancing the acquisition, organization, analysis and decision of government affair data, the government affair information resources are uniformly managed, developed and utilized according to laws and regulations and the requirements of each department, the utilization rate of the data resources can be improved, repeated construction is avoided, and the maintenance cost is reduced; and the decision-making efficiency is further improved by deeply mining government affair information resources.
The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (9)
1. A government affairs big data platform system, characterized in that the system comprises:
the data convergence subsystem is used for converging heterogeneous multi-source data;
the data asset subsystem is used for carrying out full life cycle management on the data in the data aggregation subsystem according to government affair data asset management rules;
the data service subsystem is used for determining the use mode of the data in the data aggregation subsystem;
the data management subsystem is used for managing and controlling the data quality in the data aggregation subsystem; and monitoring service operation and platform operation and maintenance in the system.
2. The system of claim 1, wherein the data access mode of the data aggregation subsystem comprises at least one of the following modes:
the database mode supports various main stream databases of MySQL, Oracle, SqlServer, Hive and/or HBase;
the file mode supports formatted files and unformatted files; through data reporting, the extraction of data in the formatted file is supported and the data is stored in a relational database; uploading and gathering the unformatted files through the files;
and the interface mode supports RESTful and Web Service interface types.
3. The system of claim 1, wherein the data aggregation subsystem is configured to:
the operation is periodically operated according to a scheduling plan to realize data aggregation;
monitoring a running process of a job using job running monitoring, the running process including at least one of: run time, run status, successfully processed data, query data, run logs, and historical run conditions;
monitoring hardware information of a server running the data aggregation subsystem using device operation monitoring, the hardware information including at least one of: CPU occupation, memory occupation and disk occupation;
and when the hardware information indicates that the server has abnormity, early warning is carried out.
4. The system of claim 1, wherein the data asset subsystem is configured to:
classifying the data in the data aggregation subsystem according to a preset resource cataloging rule to obtain a resource classification;
registering resource information corresponding to the data in the data aggregation subsystem, and establishing a data association table corresponding to the registered resource information, wherein the data association table is used for associating the resource information with the data in the data aggregation subsystem; the resource information comprises basic information and resource information items; the basic information comprises a resource name, a resource classification, a resource abstract, a resource type, a sharing mode and/or an updating period; the resource information item is used for describing the classified resource;
issuing the registered resource information and providing inquiry service of the issued resource information;
when a viewing request of target data corresponding to target resource information is received, if the sharing type of the target data is conditional sharing, a demand department sends a resource viewing demand to a resource providing department so that the resource providing department can review the resource viewing demand; the resource viewing demand comprises a resource information item, a demand reason and demand time of the target resource information;
after the resource providing department passes the examination and verification of the resource viewing requirement, recording in a data management department to form a three-party protocol; after the examination and the record are passed, allowing a demand department to use target data corresponding to the target resource information; and the demand department calls data according to the sharing mode corresponding to the target data.
5. The system of claim 1, wherein the data services subsystem is configured to:
receiving an interface service registered and issued by an interface providing department, wherein the interface service is used for calling external data or feeding back a result according to an input condition, and the interface service comprises at least one of the following information: basic information, out-participation-parameter information, example codes and error codes;
and generating a service description document according to the interface service, and providing an export function of the service description document so as to enable a developer to refer to and call.
6. The system of claim 5, wherein the data services subsystem is further configured to:
receiving a demand application provided by a service demand department;
sending the requirement application to an application auditing department for auditing;
after the application auditing department passes the auditing, distributing an authorized password for the service demand department;
receiving a document calling request sent by a service demand department by using a corresponding authorization password;
providing the service description document for the service demand department based on the authorized password, and recording calling information of the service description document, wherein the calling information comprises at least one of the following information: calling a service name, calling an IP, calling time, calling parameters, and calling status.
7. The system of claim 5, wherein the data services subsystem is further configured to:
determining whether the service demand department is reliable or not according to a preset white list;
determining whether the calling frequency of the service description document requested by the service demand department exceeds a preset threshold value;
and when the service demand department is reliable and the calling frequency does not exceed the preset threshold value, executing the steps of providing the service description document for the service demand department based on the authorized password and recording the calling information of the service description document.
8. The system of claim 1, wherein the data governance subsystem is configured to:
setting governing standards, wherein the governing standards comprise management standards for data desensitization, data authority, data code values and data verification;
making an ETL operation and a scheduling plan of the ETL operation according to the treatment standard;
scheduling the ETL operation according to the scheduling plan so as to clean data according to the treatment standard to obtain a treatment result;
and processing the treatment result.
9. The system of claim 1, wherein the data governance subsystem is configured to:
classifying data which does not meet the governing standard into question data;
displaying the data field content and the question reason of the question data;
checking a source business database of the query data;
adjusting the related business operation of the source business database according to the checking result;
data is extracted through the ETL job.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010903094.9A CN112035438A (en) | 2020-09-01 | 2020-09-01 | Government affair big data platform system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010903094.9A CN112035438A (en) | 2020-09-01 | 2020-09-01 | Government affair big data platform system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112035438A true CN112035438A (en) | 2020-12-04 |
Family
ID=73590478
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010903094.9A Pending CN112035438A (en) | 2020-09-01 | 2020-09-01 | Government affair big data platform system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112035438A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112650745A (en) * | 2020-12-30 | 2021-04-13 | 中科环森智慧科技(苏州)有限公司 | Data management system based on unified data resource pool |
| CN113076361A (en) * | 2021-03-17 | 2021-07-06 | 国家气象信息中心(中国气象局气象数据中心) | Method for realizing massive meteorological data unified service interface based on big data |
| CN113094393A (en) * | 2021-03-16 | 2021-07-09 | 杭州数梦工场科技有限公司 | Data aggregation method and device and electronic equipment |
| CN113987077A (en) * | 2021-12-23 | 2022-01-28 | 太极计算机股份有限公司 | Data sensing and cross-link scheduling method and device based on chain code mechanism |
| CN115496428A (en) * | 2022-11-18 | 2022-12-20 | 北京融数安科技有限公司 | Industrial safety management method and system based on big data |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160364927A1 (en) * | 2015-06-15 | 2016-12-15 | Blub0X Technology Holdings, Inc. | Web-cloud hosted unified physical security system |
| CN108647217A (en) * | 2017-12-27 | 2018-10-12 | 广东智政信息科技有限公司 | Big data platform integrated management system based on safety supervision application |
| CN110781236A (en) * | 2019-10-29 | 2020-02-11 | 山西云时代技术有限公司 | Method for constructing government affair big data management system |
| CN111259006A (en) * | 2019-11-19 | 2020-06-09 | 中国科学院计算机网络信息中心 | Universal distributed heterogeneous data integrated physical aggregation, organization, release and service method and system |
-
2020
- 2020-09-01 CN CN202010903094.9A patent/CN112035438A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160364927A1 (en) * | 2015-06-15 | 2016-12-15 | Blub0X Technology Holdings, Inc. | Web-cloud hosted unified physical security system |
| CN108647217A (en) * | 2017-12-27 | 2018-10-12 | 广东智政信息科技有限公司 | Big data platform integrated management system based on safety supervision application |
| CN110781236A (en) * | 2019-10-29 | 2020-02-11 | 山西云时代技术有限公司 | Method for constructing government affair big data management system |
| CN111259006A (en) * | 2019-11-19 | 2020-06-09 | 中国科学院计算机网络信息中心 | Universal distributed heterogeneous data integrated physical aggregation, organization, release and service method and system |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112650745A (en) * | 2020-12-30 | 2021-04-13 | 中科环森智慧科技(苏州)有限公司 | Data management system based on unified data resource pool |
| CN113094393A (en) * | 2021-03-16 | 2021-07-09 | 杭州数梦工场科技有限公司 | Data aggregation method and device and electronic equipment |
| CN113094393B (en) * | 2021-03-16 | 2023-07-14 | 杭州数梦工场科技有限公司 | Data aggregation method and device and electronic equipment |
| CN113076361A (en) * | 2021-03-17 | 2021-07-06 | 国家气象信息中心(中国气象局气象数据中心) | Method for realizing massive meteorological data unified service interface based on big data |
| CN113987077A (en) * | 2021-12-23 | 2022-01-28 | 太极计算机股份有限公司 | Data sensing and cross-link scheduling method and device based on chain code mechanism |
| CN115496428A (en) * | 2022-11-18 | 2022-12-20 | 北京融数安科技有限公司 | Industrial safety management method and system based on big data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112035438A (en) | Government affair big data platform system | |
| EP3869371A1 (en) | Data consent manager | |
| US11962614B2 (en) | Techniques for cloud security monitoring and threat intelligence | |
| EP3422269A1 (en) | Centralized consent management | |
| US9892281B1 (en) | Testing using deidentified production data | |
| US9641555B1 (en) | Systems and methods of tracking content-exposure events | |
| US20090271238A1 (en) | System and method of managing a workflow within and between a criminal case management side and a criminal intelligence management side in the justice and public safety domain | |
| US20210294853A1 (en) | Predicted data use obligation match using data differentiators | |
| US11734651B2 (en) | Rendering related content prior to an event in a group-based communication interface | |
| US11477244B2 (en) | Method and system for data loss prevention management | |
| AU2018220072B2 (en) | Systems and methods to control data access and usage | |
| US8620911B2 (en) | Document registry system | |
| US20180013643A1 (en) | Determining events by analyzing stored electronic communications | |
| CN109766322A (en) | A kind of data share exchange method and system | |
| US11126445B1 (en) | Disparate data aggregation for user interface customization | |
| CN104504014A (en) | Data processing method and device based on large data platform | |
| CN112348664B (en) | Credit credit management system supporting automatic early warning | |
| US20230267387A1 (en) | Computer-Guided Corporate Relationship Management | |
| US20180349983A9 (en) | A system for periodically updating backings for resource requests | |
| CN104035939A (en) | Flexible monitoring frame with peculiar independent rule engine | |
| US8832856B2 (en) | Authority delegation for business objects | |
| CN114175577A (en) | Information barrier for sensitive information | |
| CN112835863A (en) | Processing method and processing device of operation log | |
| CN115629880A (en) | Log desensitization method, device, equipment and storage medium | |
| US20230177180A1 (en) | Automatically determining application responder groups for data privacy integration services |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |