CN114860711A - Environment-friendly data processing system and method - Google Patents
Environment-friendly data processing system and method Download PDFInfo
- Publication number
- CN114860711A CN114860711A CN202210589922.5A CN202210589922A CN114860711A CN 114860711 A CN114860711 A CN 114860711A CN 202210589922 A CN202210589922 A CN 202210589922A CN 114860711 A CN114860711 A CN 114860711A
- Authority
- CN
- China
- Prior art keywords
- data
- environmental protection
- environment
- management
- friendly
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 75
- 238000000034 method Methods 0.000 title claims abstract description 35
- 230000007613 environmental effect Effects 0.000 claims abstract description 166
- 238000013523 data management Methods 0.000 claims abstract description 62
- 238000004458 analytical method Methods 0.000 claims abstract description 33
- 230000007246 mechanism Effects 0.000 claims abstract description 8
- 230000010354 integration Effects 0.000 claims abstract description 4
- 238000007726 management method Methods 0.000 claims description 56
- 230000008569 process Effects 0.000 claims description 15
- 238000013075 data extraction Methods 0.000 claims description 14
- 238000004422 calculation algorithm Methods 0.000 claims description 11
- 230000005540 biological transmission Effects 0.000 claims description 8
- 238000006243 chemical reaction Methods 0.000 claims description 8
- 238000013524 data verification Methods 0.000 claims description 8
- 238000007405 data analysis Methods 0.000 claims description 7
- 238000013079 data visualisation Methods 0.000 claims description 6
- 238000011835 investigation Methods 0.000 claims description 6
- 238000004088 simulation Methods 0.000 claims description 6
- 238000004140 cleaning Methods 0.000 claims description 5
- 230000002159 abnormal effect Effects 0.000 claims description 4
- 238000000605 extraction Methods 0.000 claims description 4
- 230000004931 aggregating effect Effects 0.000 claims description 3
- 238000009960 carding Methods 0.000 claims description 3
- 230000006698 induction Effects 0.000 claims description 3
- 238000013507 mapping Methods 0.000 claims description 3
- 238000004806 packaging method and process Methods 0.000 claims description 3
- 238000007619 statistical method Methods 0.000 claims description 3
- 238000012800 visualization Methods 0.000 claims description 3
- 238000003672 processing method Methods 0.000 claims description 2
- 238000012544 monitoring process Methods 0.000 description 20
- 238000010276 construction Methods 0.000 description 8
- 238000013439 planning Methods 0.000 description 7
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 6
- 230000005855 radiation Effects 0.000 description 5
- 238000003860 storage Methods 0.000 description 5
- 238000013461 design Methods 0.000 description 4
- 239000003344 environmental pollutant Substances 0.000 description 4
- 238000011156 evaluation Methods 0.000 description 4
- 231100000719 pollutant Toxicity 0.000 description 4
- 239000002910 solid waste Substances 0.000 description 4
- 238000001514 detection method Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000006872 improvement Effects 0.000 description 3
- 230000002265 prevention Effects 0.000 description 3
- 239000002689 soil Substances 0.000 description 3
- 238000012795 verification Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 239000002920 hazardous waste Substances 0.000 description 2
- 239000000126 substance Substances 0.000 description 2
- 238000003911 water pollution Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000033228 biological regulation Effects 0.000 description 1
- 210000004556 brain Anatomy 0.000 description 1
- 238000009395 breeding Methods 0.000 description 1
- 230000001488 breeding effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013145 classification model Methods 0.000 description 1
- 238000013479 data entry Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000009792 diffusion process Methods 0.000 description 1
- 239000003651 drinking water Substances 0.000 description 1
- 235000020188 drinking water Nutrition 0.000 description 1
- 239000000428 dust Substances 0.000 description 1
- 239000000383 hazardous chemical Substances 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 244000144972 livestock Species 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 244000144977 poultry Species 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000004055 radioactive waste management Methods 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 239000002352 surface water Substances 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 239000002351 wastewater Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/26—Visual data mining; Browsing structured data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/906—Clustering; Classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/10—Services
- G06Q50/26—Government or public services
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Tourism & Hospitality (AREA)
- Educational Administration (AREA)
- Development Economics (AREA)
- Health & Medical Sciences (AREA)
- Economics (AREA)
- General Health & Medical Sciences (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- General Business, Economics & Management (AREA)
- Quality & Reliability (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention relates to an environment-friendly data processing system and method, which solve the problems of low quality of environment-friendly data, lack of uniform data management and safe data environment and the like which generally exist at present. An environment-friendly data processing system comprising: the system establishing module is used for establishing a corresponding database according to a resource system of the environmental protection data; the data integration module is used for acquiring environment-friendly data and storing the environment-friendly data into a corresponding database according to a resource system of the environment-friendly data; the data management module is used for carrying out data management and data management on the stored environment-friendly data; the shared exchange module is used for collecting the environment-friendly data subjected to data management and data management to a shared resource pool through a data shared exchange mechanism; and the analysis publishing module is used for analyzing and processing the environmental protection data in the shared resource pool and calling the environmental protection data after analysis and processing according to the service requirement.
Description
Technical Field
The present invention relates to the field of environmental protection technologies, and in particular, to an environmental protection data processing system and method.
Background
At present, the environmental protection industry needs to continuously promote the improvement of an environmental protection business management system and the improvement of management capability, and the environmental protection business management system is centered around the special fields of water ecological environment, atmospheric environment, solid waste and chemicals, soil ecological environment, marine ecological environment, enterprise pollution sources and the like, and environment-friendly key business scenes such as focused water pollution control, atmospheric pollution control, solid waste hazardous waste full life cycle supervision, noise monitoring, soil plot investigation, pollution source traceability management, near shore sea area monitoring and the like are created on the basis of a new generation of information technology such as big data, AI, IOT, 5G, remote sensing, edge cloud, unmanned aerial vehicle, unmanned ship and the like on the basis of an environmental protection intelligent monitoring system, so as to create an environment-friendly intelligent efficient decision support service and achieve the overall goal of real-time 'mining, storage, taking, using and adjusting'. In actual business of the environmental protection industry, the problems that a plurality of established information systems are constructed in a dispersed manner, environmental protection data cannot be collected and processed uniformly, uniform data management and safe data environment are lacked and the like generally exist.
Disclosure of Invention
The present invention is directed to overcome at least one of the above-mentioned drawbacks of the prior art, and to provide an environment-friendly data processing system and method, so as to solve the problems of low quality of environment-friendly data, lack of uniform data management and secure data environment, and the like.
On one hand, the invention adopts the technical scheme that the environment-friendly data processing system comprises:
the system establishing module is used for establishing a corresponding database according to a resource system of the environmental protection data;
the data integration module is used for acquiring environment-friendly data and storing the environment-friendly data into a corresponding database according to a resource system of the environment-friendly data;
the data management module is used for carrying out data management and data management on the stored environment-friendly data;
the shared exchange module is used for collecting the environment-friendly data subjected to data management and data management to a shared resource pool through a data shared exchange mechanism;
and the analysis publishing module is used for analyzing and processing the environmental protection data in the shared resource pool and calling the environmental protection data after analysis and processing according to the service requirement.
Further, the establishing of the corresponding database according to the resource system of the environmental protection data includes:
the resource system of the environmental protection data is divided into an internal data system of the environmental system and an external data system of the environmental system according to data sources,
dividing an internal data system of the environment system into a basic service data system and a subject data system according to the environment management service, wherein the subject data system is formed by induction processing based on the basic service data system;
dividing an external data system of the environment system into an external department data system and an internet public data system according to an environment data source;
and establishing a corresponding database based on the basic service data system, the subject data system, the external department data system and the Internet public data system.
Further, the acquiring environmental protection data includes:
carrying out data resource investigation on the environmental protection data, and carding and checking the service information of the environmental protection service to generate a checking result;
generating a data ledger according to the checking result, and acquiring original environment-friendly data according to the data ledger;
and processing the original environment-friendly data based on a preset error processing strategy, breakpoint continuous transmission and breakpoint continuous extraction to obtain environment-friendly data.
Further, the data management module includes:
the data governance service unit is used for carrying out data governance on the stored environmental protection data, and the data governance comprises one or more of the following processes:
data extraction processing: extracting environmental protection data required by a target data source system from the data source system;
data verification processing: in the data extraction processing, comparing and analyzing the environmental protection data of the data source system and the target data original system to obtain abnormal error information generated in the data extraction processing;
and (3) data cleaning treatment: reconstructing the environmental protection data, deleting repeated and invalid environmental protection data, and correcting errors existing in the environmental protection data;
and (3) data conversion processing: converting the data type of the environmental protection data;
and (3) data compensation processing: and performing completion treatment on the missing environmental protection data based on the historical environmental protection data.
Further, the data management module includes:
the data management service unit is used for performing data management on the stored environmental protection data, and the data management comprises one or more of the following management:
data asset management: performing asset pre-standard management and asset statistical analysis on the environmental protection data from the perspective of assets;
and (3) data quality management: based on a preset management rule, checking the quality of the environmental protection data;
resource catalog management: and establishing a corresponding resource directory system according to the source and/or type of the information resource of the environmental protection data, and classifying the environmental protection data according to the resource directory system.
Further, the establishing of the corresponding resource directory system according to the source and/or type of the information resource of the environmental protection data includes:
and mapping the information resources of the environmental protection data into a resource directory system according to the source and/or type of the information resources of the environmental protection data, automatically filling in metadata information, and establishing the relevance of the information resources and the resource directory system.
Further, the aggregating the environmental protection data subjected to data governance and data management to a shared resource pool includes:
the environmental protection data after data management and data management are sequenced and coded, the characteristics of the environmental protection data are extracted, and the environmental protection data are subjected to relevance analysis according to the characteristics and then are collected to a shared resource pool.
Further, the shared switching module includes:
the data sharing service unit is used for packaging the environment-friendly data subjected to data management and providing various data sharing services through a uniform service interface;
the data access service unit is used for opening a corresponding data access interface according to the received access application;
and the data exchange service unit is used for providing data exchange transmission channels for various service information systems.
Further, the analysis publishing module comprises:
the system comprises a data intelligent unit, a data analysis unit and a data analysis unit, wherein the data intelligent unit comprises an algorithm supermarket, an environment-friendly service scene model and a CIM (common information model), and the algorithm supermarket, the environment-friendly service scene model and the CIM are respectively used for performing algorithm processing, scene application simulation and CIM simulation on environment-friendly data;
and the data publishing unit is used for containing a data visualization component and an analysis report module, and the data visualization component and the analysis report module are respectively used for performing visualization operation on the environmental protection data and analyzing the environmental protection data to form an analysis report.
On the other hand, the invention adopts the technical scheme that the method for processing the environment-friendly data comprises the following steps:
establishing a corresponding database according to a resource system of the environmental protection data;
acquiring environment-friendly data, and storing the environment-friendly data into a corresponding database according to a resource system of the environment-friendly data;
carrying out data governance and data management on the stored environmental protection data;
collecting the environment-friendly data subjected to data treatment and data management to a shared resource pool through a data sharing exchange mechanism;
analyzing and processing the environmental protection data in the shared resource pool, and calling the environmental protection data after analysis and processing according to the service requirement.
Compared with the prior art, the invention has the beneficial effects that: the environment-friendly data processing system is provided, so that the common management of environment-friendly data in one system is realized, and the problems of low data quality, lack of unified data management, lack of safe data environment and the like in the environment-friendly industry are solved.
Drawings
FIG. 1 is an architecture diagram of an environment-friendly data processing system according to the present invention.
FIG. 2 is a flowchart illustrating an exemplary environment-friendly data resource sharing scheduling process.
Detailed Description
The drawings are only for purposes of illustration and are not to be construed as limiting the invention. For a better understanding of the following embodiments, certain features of the drawings may be omitted, enlarged or reduced, and do not represent the size of an actual product; it will be understood by those skilled in the art that certain well-known structures in the drawings and descriptions thereof may be omitted.
The environment-friendly data processing system is constructed based on environment-friendly data standard specifications, is used for standardizing related data operations such as data acquisition, data entry, data processing, data storage, data internal sharing and external exchange, and can meet the requirements of running and managing an ecological environment information network and an application system orderly, stably and reliably only by following and implementing related standards specified by the international, domestic and government data brains of information technology.
As shown in fig. 1, which is an architecture diagram of an environmental protection data processing system according to an embodiment of the present invention, the environmental protection processing system includes: a system establishing module 10, a data integrating module 20, a data managing module 30, a sharing exchanging module 40, and an analyzing and publishing module 50, wherein,
the system establishing module 10 is used for establishing a corresponding database according to the resource system of the environmental protection data;
preferably, the establishing of the corresponding database according to the resource system of the environmental protection data includes:
the resource system of the environmental protection data is divided into an internal data system of the environmental system and an external data system of the environmental system according to data sources,
dividing an internal data system of the environment system into a basic service data system and a subject data system according to the environment management service, wherein the subject data system is formed by induction processing based on the basic service data system;
dividing an external data system of the environment system into an external department data system and an internet public data system according to an environment data source;
and establishing a corresponding database based on the basic service data system, the subject data system, the external department data system and the Internet public data system.
It should be noted that the step of establishing the resource system of the environmental protection data includes: planning data resources, designing a data system and constructing a database system;
for data resource planning, the following specific functions are available in practical applications: defining the data range of the data resource plan: all the environmental protection related member units have related business data, part of administrative office data and some external application data; defining the scope of the functional domain of the data resource planning: on the basis of the overall understanding of the data, the result of the business requirement analysis and the information classification are combined with the code standard for determination, and the method comprises the following steps: the method comprises the following steps of 24 environmental services, namely, pollution discharge license, pollution discharge right transaction, environmental statistics, pollution source supervision monitoring, construction project environmental evaluation approval, construction project completion acceptance, pollution source online monitoring, pollution discharge declaration charging, mobile law enforcement, administrative punishment, petition complaint, environmental information disclosure, enterprise credit evaluation, environmental emergency, solid waste, radiation license, enterprise self-monitoring, motor vehicle management, environmental quality, environmental statistics and the like, and further comprises external metadata: data element sets such as data of an industrial and commercial department, data of a transportation department, data of a statistical department, data of a water conservancy department, data of an agricultural department, data of a homeland planning department, data of a health planning department, data of a tax department, data of a security inspection department, data of a public security department and the like; the specific planning is carried out in a single functional domain according to a data resource planning method: establishing a theme model of each dimension such as a pollution source theme domain, an environment quality theme domain, a comprehensive theme domain and the like according to the individual function domain, and defining and disclosing various data of the business related to each analysis object and the relation among the data; forming a data element set of the environment-friendly data resource center: on the basis of the arrangement of the data elements of each functional domain, a complete and clear classified and non-repetitive data element set is formed.
For the data system design, the resource classification of the environmental protection data is firstly combed, and then the resource system design of the environmental protection data is carried out. The resource classification of the environmental protection data can be specifically classified according to three modes of a data source, a service range and a data type, and when the resource classification is classified according to the data source: the method comprises the following steps of dividing the data into environmental system internal data and environmental system external data, wherein the internal data comprises contents such as pollution source management, environmental quality, nuclear and radiation, ecological environment, environmental protection capability, emergency management and the like, and the external data comprises contents such as social and economic data, hydrological weather, urban construction and the like which are closely related to the environment; when classified according to service scope: the method comprises the following steps of dividing the data into service data and auxiliary data, wherein the service data are generated by different service management processes (statistics, monitoring, supervision and the like) and reflect the condition of service management, and the auxiliary data are used for providing diversified support for the application of the service data, such as spatial data, public coded data and the like; when classified according to data type: the method comprises the steps of dividing the data into structured data, unstructured data, semi-structured data and time sequence data, wherein the structured data can be represented and stored by a uniform structure, such as statistical data; unstructured data cannot be represented numerically or with a uniform structure, such as documents, images, sound, etc.; the semi-structured data is captured from the Internet; the time sequence data has the characteristic of strong time sequence and has higher real-time requirement.
The embodiment mainly aims at the aspect of data source to carry out resource system design on the environmental protection data, and the environmental protection data is divided into environmental system internal data and environmental system external data. Designing a resource system aiming at internal data of an environment system, wherein the internal data of the environment system is divided into two categories according to related environment management services, wherein one category is basic service data, and the other category is subject data, so that a basic service data system and a subject data system are established; basic service data are generally dispersed, reflect the problems in a certain service management field, need to be processed and integrated, realize the standardization of the data, solve the problems of inconsistent data statistical calibers, multiple sources, conflict and redundancy, improve the accuracy, reliability, consistency and usability of the data and provide a unique, real and credible data view; the theme data is formed by summarizing and integrating basic service data on a certain level according to the requirement of data analysis. The subject is an abstract concept, which corresponds to an analysis object related to a macro analysis field in business application, and carries out a complete and consistent description on the data of the analysis object at a higher level, and defines and reveals various data of businesses related to various analysis objects and the relation between the data. The external data access capability is reserved, the resource system design is carried out on the external data of the environment system, the external data of the environment system are divided according to the data source of the external data of the environment system, the external data of the environment system comprise external department data and internet public data, and therefore an external department data system and an internet public data system are established. Notably, the external department data obtained includes: natural resources, urban and rural construction of housing, transportation, industry and commerce, tax, water conservancy, rural agriculture, forestry, meteorological data of departments and units.
The data integration module 20 is configured to acquire environmental protection data, and store the environmental protection data in a corresponding database according to a resource system of the environmental protection data;
preferably, the acquiring environmental protection data includes:
carrying out data resource investigation on the environmental protection data, and carding and checking the service information of the environmental protection service to generate a checking result;
generating a data ledger according to the checking result, and acquiring original environment-friendly data according to the data ledger;
and processing the original environment-friendly data based on a preset error processing strategy, breakpoint continuous transmission and breakpoint continuous extraction to obtain environment-friendly data.
Specifically, according to one or more information in the environment-friendly service type, the data type and the construction condition of an application system and a database, data resource investigation of environment-friendly data is carried out, the information generated and used by the service is combed and checked to generate a checking result, and a data ledger is generated according to the checking result; according to the data ledger, acquiring original environment-friendly data through multiple channels such as manual execution of import packets, regular synchronization of databases, automatic warehousing of data interfaces and the like; processing the original environment-friendly data through a preset error processing strategy, breakpoint continuous transmission and breakpoint continuous extraction, and connecting a required multi-source database to capture and store data in real time; establishing different business table structure associations and associations among different system business data for the processed environment-friendly data, and performing relational storage, semi-structured storage, time-sequence storage and object storage on the environment-friendly data according to data characteristics.
In this embodiment, an environmental protection database is established according to a resource system and the obtained environmental protection data, and the environmental protection database includes an ecological environment quality information database, a pollution source information database, an environment management service information database, a geospatial database, an environmental protection video database, and an external information database. The method specifically comprises the following steps: (1) the ecological environment quality information database can be used for combing environment quality data, environment quality reports, environment functional divisions, monitoring/detection management and other environment quality related information, and specifically comprises water/atmosphere/sound/soil/countryside/radiation and other environment quality data, environment condition bulletin, environment white paper, environment monitoring annual newspaper/quarter newspaper/monthly newspaper, pollution accident emergency environment monitoring express newspaper, surface water environment/atmosphere environment/noise environment/ecology/drinking water source area and other functional divisions, a monitoring/detection management system, laboratory management, monitoring/detection instrument management and other data. (2) The pollution source information database can be used for combing the basic information of pollution sources, non-point sources, emission lists, pollution source investigation statistics, pollution source monitoring and control, pollutants and other related information of the pollution sources, and specifically comprises data such as industrial/agricultural/living/moving/flying dust pollution source basic data, rural living, livestock and poultry breeding, cultivated land, waste water emission lists, atmospheric emission lists, environment statistics, pollution source general survey, pollution source online monitoring, pollution source supervision monitoring, enterprise self-monitoring, pollutant types and properties, pollutant removal methods and the like. (3) The environment management service database can be used for combing environment management service related information such as environment examination and approval, environment supervision, pollution prevention and control, nuclear and radiation safety management, environment pollution accident and emergency management, environment public participation and the like, and specifically comprises data such as environment influence evaluation management, completion acceptance, pollution discharge license management, petition complaint, mobile law enforcement, administrative penalty, enterprise credit evaluation, water pollution prevention and control, atmospheric pollution prevention and control, radiation safety management, radioactive waste management, pollution accident management, risk source management, emergency command scheduling, hazardous chemical substance data, emergency data management, government information disclosure and the like. (4) The geographic spatial database can be used for combing service layers existing in an environmental protection hall, and comprises environmental public spatial data, basic geographic data and remote sensing data, wherein the environmental public spatial data comprise special layers of environmental quality special subjects, pollution source special subjects, environmental supervision special subjects, ecological protection and functional area special subjects, land utilization and the like. (5) The environment-friendly video database can integrate and cover the environment-friendly videos of monitoring points such as key watersheds, key industrial parks/industrial parks, environment receptor sensitive areas, heavy pollution sources, solid waste and hazardous waste transfer and the like of a whole province through an e-government network and an environment-friendly special network, and tries to coordinate and access video resources of departments such as water conservancy, water affairs, city management, municipal administration, public security, maritime affairs, traffic, marine fishery, weather and the like, so that the real-time monitoring of environmental accident emergency is realized, and the related work requirements of ecological environment resource protection and management are met.
And the data management module 30 is used for performing data management and data management on the stored environmental protection data. The method specifically comprises the following steps: the data governance service unit is used for carrying out data governance on the stored environment-friendly data, and the data governance comprises the following steps:
data extraction processing: extracting environmental protection data required by a target data source system from the data source system;
data verification processing: in the data extraction processing, comparing and analyzing the environmental protection data of the data source system and the target data original system to obtain abnormal error information generated in the data extraction processing;
and (3) data cleaning treatment: reconstructing the environmental protection data, deleting repeated and invalid environmental protection data, and correcting errors existing in the environmental protection data;
and (3) data conversion processing: converting the data type of the environmental protection data;
and (3) data compensation processing: and performing completion treatment on the missing environmental protection data based on the historical environmental protection data.
Specifically, the data management comprises one or more of data extraction processing, data verification processing, data cleaning processing, data conversion processing, data compensation processing, data assimilation processing, data examination and loading, so that heterogeneous data of different sources related to pollution sources, environmental quality and the like are efficiently integrated, and the uniqueness of the environment-friendly data and the relevance of the original data are guaranteed.
The data extraction process is to extract data required by a target data source system from the data source system.
The data verification processing is to compare and analyze data between a data source and a target database in the data extraction process so as to further analyze, find and solve abnormal error information possibly generated in the data extraction process, verify the environmental protection data by using the data verification processing, analyze fine-grained dimensionality of a verification object, support file-level verification and record-level verification, and filter data verification contents if the data verification contents have information such as type, length, whether the data are empty, precision, range, format and the like, and only correct data can be continuously used if the data are not in accordance; and outputting error data, wherein the error data comprises information such as error reasons and error field sequence numbers, and statistical information such as related error types and numbers is bound to the process variables, so that the subsequent nodes can be judged and used.
The data cleaning process is to reconstruct the data, delete duplicate, invalid, etc. data, correct existing errors, and provide data consistency.
The data conversion processing is to convert the related data types through a standardized and customized conversion interface, and comprises the following steps: type conversion, field splitting, merging, string processing, date conversion, and the like.
The data compensation processing is used for performing compensation processing on the missing data to ensure the data integrity, lays a foundation for business modeling or algorithm calculation at the next stage, predicts missing field data as a target variable based on the existing historical collected data to obtain the most possible compensation value, and if the column with the missing value is a numerical variable, performs compensation by adopting a regression model; and if the variable is the classification variable, completing by adopting a classification model.
The data assimilation processing automatically adjusts the pollutant diffusion model track by continuously fusing direct or indirect monitoring information of different sources and different resolutions which are discretely distributed in time and space through a data assimilation algorithm so as to improve the estimation precision of the environment dynamic model state and improve the model pollution prediction capability.
The data management module 30 further includes: a data management service unit, configured to perform data management on the stored environmental protection data, where the data management includes:
data asset management: performing asset pre-standard management and asset statistical analysis on the environmental protection data from the perspective of assets;
and (3) data quality management: based on a preset management rule, checking the quality of the environmental protection data; the method helps users locate data quality problems (such as missing, repeating, extreme value abnormity, monotonicity abnormity and the like), and track and discover system problems, specifically including rule management, data scanning and problem discovery;
resource catalog management: and establishing a corresponding resource directory system according to the source and/or type of the information resource of the environmental protection data, and classifying the environmental protection data according to the resource directory system. Wherein, according to the source and/or type of the information resource of the environmental protection data, establishing a corresponding resource directory system, comprising: and mapping the information resources of the environmental protection data into a resource directory system according to the source and/or type of the information resources of the environmental protection data, automatically filling in metadata information, and establishing the relevance of the information resources and the resource directory system.
In addition, the functions of the data management module further include: common code management, data dictionary management. Common code management is used for: various public codes in a public code base are added, deleted, changed and checked, meanwhile, the batch loading of public code files is supported, and the processing of different public code data files can be realized through configuration; the method comprises 63 types of standard codes of environment-related countries and industry technical regulations and other standard codes related to pollution sources and environment quality; all business data enter the system, and the related environment codes need to be uniformly matched and standardized so as to ensure the uniformity and accuracy of the system data. The data dictionary management is for: the method comprises the steps of maintaining a data dictionary type, maintaining a data dictionary value, retrieving data dictionary information, and managing data dictionary configuration (including information validity period, application range and the like); the data dictionary is used for describing an environment information database system structure and recording information such as sources, descriptions, relationships with other data, purposes and formats of ecological environment data.
And the shared exchange module 40 is configured to aggregate the environmental protection data subjected to data governance and data management to a shared resource pool through a data shared exchange mechanism.
The step of aggregating the environmental protection data subjected to data governance and data management to a shared resource pool comprises the following steps:
the environmental protection data after data management and data management are sequenced and coded, the characteristics of the environmental protection data are extracted, and the environmental protection data are subjected to relevance analysis according to the characteristics and then are collected to a shared resource pool.
The shared switching module 40 includes:
the data sharing service unit is used for packaging the environment-friendly data subjected to data management and providing various data sharing services through a uniform service interface; the method comprises the full-flow management of data service registration, auditing, release, use and monitoring, provides service invocation in a standard RESTful mode, and provides platform internal data query and data release.
The data access service unit is used for opening a corresponding data access interface according to the received access application; a user may be provided with a spreadsheet access service, a folder access service, and a real-time data interface channel access service.
And the data exchange service unit is used for providing data exchange transmission channels for various service information systems. The data exchange service is not limited to static data transmission, can realize cross-region and cross-system data exchange and unified scheduling management of data resources, and can also exchange data with application systems of other mechanisms.
As shown in fig. 2, a data resource sharing scheduling flow applied in the environment-friendly data sharing exchange is provided. The environment-friendly data resource sharing and dispatching process comprises an environment-friendly data demand side, a data service management side, an environment-friendly data source side and an environment-friendly data source system construction side. The specific process of the environment-friendly data resource sharing scheduling comprises the following steps: firstly, the environmental protection data demand side puts forward the data demand, firstly, the system will give the usage demand to the management side for compliance check, if the usage demand put forward by the user is compliant, reasonable and legal, then the data source system construction party is given specific analysis according to the flow, the specific analysis is carried out on the data source system construction party, the number list, the data amount, the data current situation and the like, then the system sends the result obtained by the analysis to the data management party for auditing, whether the data are conditionally shared or not is determined, if the data are conditionally shared, performing letter sending coordination according to a preset resource application flow, performing cataloging, hooking and pushing on data according to the flow, confirming the completion condition of the data by a data management party, if the catalog hanging is completed, the data source system constructor can perform service implementation works such as data preparation and data extraction, in the process, the data management party confirms that the data is successfully exported to the user number demand party through the system; then according to the flow, the data management party can make a return visit plan and evaluate the effectiveness of the application. The environment-friendly data scheduling and sharing process provided by the embodiment can standardize and promote the work of environment protection and data resource sharing, so that the system can orderly process the requirements of users on data, and the efficiency of processing data is improved. .
And the analysis publishing module 50 is configured to analyze and process the environmental protection data in the shared resource pool, and call the environmental protection data after analysis and processing according to a service requirement. The method specifically comprises the following steps:
the system comprises a data intelligent unit, a data analysis unit and a data analysis unit, wherein the data intelligent unit comprises an algorithm supermarket, an environment-friendly service scene model and a CIM (common information model), and the algorithm supermarket, the environment-friendly service scene model and the CIM are respectively used for performing algorithm processing, scene application simulation and CIM simulation on environment-friendly data;
and the data publishing unit is used for containing a data visualization component and an analysis report module, and the data visualization component and the analysis report module are respectively used for performing visualization operation on the environmental protection data and analyzing the environmental protection data to form an analysis report.
The invention also provides another embodiment, an environmental protection data processing method, which comprises the following steps:
establishing a corresponding database according to a resource system of the environmental protection data;
acquiring environmental protection data, and storing the environmental protection data into a corresponding database according to a resource system of the environmental protection data;
carrying out data governance and data management on the stored environmental protection data;
collecting the environment-friendly data subjected to data treatment and data management to a shared resource pool through a data sharing exchange mechanism;
analyzing and processing the environmental protection data in the shared resource pool, and calling the environmental protection data after analysis and processing according to the service requirement.
It should be understood that the above-mentioned embodiments of the present invention are only examples for clearly illustrating the technical solutions of the present invention, and are not intended to limit the specific embodiments of the present invention. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention claims should be included in the protection scope of the present invention claims.
Claims (10)
1. An environment-friendly data processing system, comprising:
the system establishing module is used for establishing a corresponding database according to a resource system of the environmental protection data;
the data integration module is used for acquiring environment-friendly data and storing the environment-friendly data into a corresponding database according to a resource system of the environment-friendly data;
the data management module is used for carrying out data management and data management on the stored environment-friendly data;
the shared exchange module is used for collecting the environment-friendly data subjected to data management and data management to a shared resource pool through a data shared exchange mechanism;
and the analysis publishing module is used for analyzing and processing the environmental protection data in the shared resource pool and calling the environmental protection data after analysis and processing according to the service requirement.
2. The system of claim 1, wherein the creating of the corresponding database based on the resource hierarchy of the environmental protection data comprises:
the resource system of the environmental protection data is divided into an internal data system of the environmental system and an external data system of the environmental system according to data sources,
dividing an internal data system of the environment system into a basic service data system and a subject data system according to the environment management service, wherein the subject data system is formed by induction processing based on the basic service data system;
dividing an external data system of the environment system into an external department data system and an internet public data system according to an environment data source;
and establishing a corresponding database based on the basic service data system, the subject data system, the external department data system and the Internet public data system.
3. The system of claim 1, wherein the obtaining environmental protection data comprises:
carrying out data resource investigation on the environmental protection data, and carding and checking the service information of the environmental protection service to generate a checking result;
generating a data ledger according to the checking result, and acquiring original environment-friendly data according to the data ledger;
and processing the original environment-friendly data based on a preset error processing strategy, breakpoint continuous transmission and breakpoint continuous extraction to obtain environment-friendly data.
4. The system of claim 1, wherein the data management module comprises:
the data governance service unit is used for carrying out data governance on the stored environmental protection data, and the data governance comprises one or more of the following processes:
data extraction processing: extracting environmental protection data required by a target data source system from the data source system;
data verification processing: in the data extraction processing, comparing and analyzing the environmental protection data of the data source system and the target data original system to obtain abnormal error information generated in the data extraction processing;
and (3) data cleaning treatment: reconstructing the environmental protection data, deleting repeated and invalid environmental protection data, and correcting errors existing in the environmental protection data;
and (3) data conversion processing: converting the data type of the environmental protection data;
and (3) data compensation processing: and performing completion treatment on the missing environmental protection data based on the historical environmental protection data.
5. The system of claim 1, wherein the data management module comprises:
the data management service unit is used for performing data management on the stored environmental protection data, and the data management comprises one or more of the following management:
data asset management: performing asset pre-standard management and asset statistical analysis on the environmental protection data from the perspective of assets;
and (3) data quality management: based on a preset management rule, checking the quality of the environmental protection data;
resource catalog management: and establishing a corresponding resource directory system according to the source and/or type of the information resource of the environmental protection data, and classifying the environmental protection data according to the resource directory system.
6. The system of claim 5, wherein the establishing a corresponding resource directory system according to the source and/or type of the information resource of the environmental protection data comprises:
and mapping the information resources of the environmental protection data into a resource directory system according to the source and/or type of the information resources of the environmental protection data, automatically filling in metadata information, and establishing the relevance of the information resources and the resource directory system.
7. The system of claim 1, wherein the aggregating the environmental protection data subjected to data governance and data management into a shared resource pool comprises:
the environmental protection data after data management and data management are sequenced and coded, the characteristics of the environmental protection data are extracted, and the environmental protection data are subjected to relevance analysis according to the characteristics and then are collected to a shared resource pool.
8. The system of claim 1, wherein the shared switching module comprises:
the data sharing service unit is used for packaging the environment-friendly data subjected to data management and providing various data sharing services through a uniform service interface;
the data access service unit is used for opening a corresponding data access interface according to the received access application;
and the data exchange service unit is used for providing data exchange transmission channels for various service information systems.
9. The system of claim 1, wherein the analytics publishing module comprises:
the system comprises a data intelligent unit, a data analysis unit and a data analysis unit, wherein the data intelligent unit comprises an algorithm supermarket, an environment-friendly service scene model and a CIM (common information model), and the algorithm supermarket, the environment-friendly service scene model and the CIM are respectively used for performing algorithm processing, scene application simulation and CIM simulation on environment-friendly data;
and the data publishing unit is used for containing a data visualization component and an analysis report module, and the data visualization component and the analysis report module are respectively used for performing visualization operation on the environmental protection data and analyzing the environmental protection data to form an analysis report.
10. An environmental protection data processing method, characterized by comprising the steps of:
establishing a corresponding database according to a resource system of the environmental protection data;
acquiring environment-friendly data, and storing the environment-friendly data into a corresponding database according to a resource system of the environment-friendly data;
carrying out data governance and data management on the stored environmental protection data;
collecting the environment-friendly data subjected to data treatment and data management to a shared resource pool through a data sharing exchange mechanism;
analyzing and processing the environmental protection data in the shared resource pool, and calling the environmental protection data after analysis and processing according to the service requirement.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210589922.5A CN114860711A (en) | 2022-05-26 | 2022-05-26 | Environment-friendly data processing system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210589922.5A CN114860711A (en) | 2022-05-26 | 2022-05-26 | Environment-friendly data processing system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114860711A true CN114860711A (en) | 2022-08-05 |
Family
ID=82641383
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210589922.5A Pending CN114860711A (en) | 2022-05-26 | 2022-05-26 | Environment-friendly data processing system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114860711A (en) |
-
2022
- 2022-05-26 CN CN202210589922.5A patent/CN114860711A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112699175B (en) | Data management system and method thereof | |
| CN110852601B (en) | Big data application method and system for environmental monitoring law enforcement decision | |
| CN107945086A (en) | A kind of big data resource management system applied to smart city | |
| CN109542967B (en) | Smart city data sharing system and method based on XBRL standard | |
| CN111459908A (en) | Multi-source heterogeneous ecological environment big data processing method and system based on data lake | |
| Kjærgaard et al. | Current practices and infrastructure for open data based research on occupant-centric design and operation of buildings | |
| CN114066211B (en) | Watershed water environment management big data platform | |
| CN104572615A (en) | Method and system for on-line case investigation processing | |
| CN112231333A (en) | Ecological environment data sharing and exchanging method and system | |
| Catlett et al. | Plenario: An Open Data Discovery and Exploration Platform for Urban Science. | |
| CN111061679B (en) | Method and system for rapid configuration of technological innovation policy based on rete and drools rules | |
| CN113626648B (en) | Water conservancy data processing system, method and storage medium | |
| KR100676693B1 (en) | A Patent Management Portal Service System | |
| CN113868318A (en) | Atmospheric environment comprehensive data acquisition and sharing system | |
| CN113901042A (en) | Ecological environment data dynamic activity level library and terminal | |
| Lehtonen et al. | Healthy mistrust or complacent confidence? Civic vigilance in the reporting by leading newspapers on nuclear waste disposal in Finland and France | |
| CN113010578B (en) | Community data analysis method and device, community intelligent interaction platform and storage medium | |
| Radermacher et al. | Trends in environmental information processing | |
| CN114860711A (en) | Environment-friendly data processing system and method | |
| CN111597332A (en) | Fiscal encyclopedia application system | |
| CN114066077B (en) | Environmental sanitation risk prediction method based on emergency event space warning sign analysis | |
| Elliot et al. | Data environment analysis and the key variable mapping system | |
| Coupette et al. | Legal hypergraphs | |
| Liu | Intelligent Community Management System Based on Big Data Technology | |
| CN117217412B (en) | Waste free urban construction management system based on resource utilization |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |