CN113901042A - Ecological environment data dynamic activity level library and terminal - Google Patents
Ecological environment data dynamic activity level library and terminal Download PDFInfo
- Publication number
- CN113901042A CN113901042A CN202111502124.6A CN202111502124A CN113901042A CN 113901042 A CN113901042 A CN 113901042A CN 202111502124 A CN202111502124 A CN 202111502124A CN 113901042 A CN113901042 A CN 113901042A
- Authority
- CN
- China
- Prior art keywords
- data
- metadata
- management unit
- ecological environment
- area
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24568—Data stream processing; Continuous queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/285—Clustering or classification
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Quality & Reliability (AREA)
- Computational Linguistics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The application relates to the field of data processing, and discloses an ecological environment data dynamic activity level library and a terminal. The horizontal library acquires ecological environment data through the data acquisition and input module, achieves cleaning of the ecological environment data according to the data standard management module to obtain a complete data set, and shares the cleaned complete data set through the external data service module. The method and the system can integrate scattered, disordered and non-uniform ecological environment data, provide a database with complete and standard data for an external system needing to use the ecological environment data, and improve the utilization rate of the ecological environment data.
Description
Technical Field
The application relates to the technical field of data processing, in particular to a dynamic activity level library and a terminal for ecological environment data.
Background
Under the promotion of the development of cloud computing, mobile internet and the like, the idea of big data is introduced into an environmental data center, and the method is a brand-new environment-friendly industry data solution.
Environmental protection departments, housing and urban and rural construction bureaus, natural resource and planning bureaus, urban management bureaus, meteorological bureaus, public security bureaus, agricultural and rural bureaus and the like all around the country can acquire massive ecological environment data each year. Through big data acquisition, management and value mining analysis, the convergence and interactive sharing of massive environmental resources are realized, the information which seems to have no correlation and fragmentation among the environmental management and reflects the surface phenomenon of a certain aspect of the problem is subjected to correlation analysis, the trend, the problem finding and the law grasping are found out, the pollutant discharge condition is stated, the current situation and the change trend of the environmental quality are stated, and the environment management department 'speaks data, manages the data and makes a decision by using the data'.
The problem of difficult collection is caused by more types of data related to pollution sources in the ecological environment, such as HTTP and WebService interface data, Excel templated data, PDF and Word document data, manual filling data, database data and the like. Moreover, the ecological environment data also has a data island problem, and the data are stored independently in different departments and are isolated from each other, which also causes the problem that complete data cannot be acquired. Therefore, the utilization rate of the ecological environment data is reduced due to the reasons of difficult data acquisition, incomplete acquired data and the like.
Disclosure of Invention
The dynamic activity level library and the terminal for the ecological environment data can integrate scattered, disordered and standard non-uniform data together, and provide a database with complete and standard data for an external system needing to use the ecological environment data.
The technical scheme adopted by the application is as follows:
the system comprises an ecological environment data dynamic activity level library, a data management module and an external data service module, wherein the level library comprises a data acquisition and input module, a data standard management module and an external data service module;
the data acquisition and input module comprises a data acquisition unit, an acquisition and input metadata management unit, an adapter management unit and a message bus management unit; the data acquisition unit is used for acquiring ecological environment data; the collection and entry metadata management unit is used for constructing metadata objects and basic components of the metadata objects; the adapter management unit is used for mapping an application data structure of an external system to a data structure of an ODS original data area; the message bus management unit is used for subscribing key nodes generated in the data acquisition process, wherein the key nodes are generated during data collection, whole data processing and data persistence;
the data standard management module comprises a standardized processing unit, an ODS original data area management unit, a data storage area management unit, a metadata area management unit and a service data area management unit; the standardized processing unit is used for carrying out standardized processing on the ecological environment data and carrying out unified organization and comprehensive management on different data in the ecological environment data; the ODS original data area management unit is used for storing the data acquired by the data acquisition and input module into the ODS original data area; the data storage area management unit is used for converting and classifying the data which are uniformly organized and comprehensively managed, and storing the converted and classified data into the data storage area; the metadata area management unit is used for converting and classifying the data which are uniformly organized and comprehensively managed, and storing the converted and classified data into a metadata area; the business data area management unit is used for converting and classifying the data which are uniformly organized and comprehensively managed, and storing the converted and classified data into the business data area;
the external data service module is used for checking and extracting the data of the service data area, wherein the data of the service data area is corresponding data generated according to a service scene.
Further, the collection entry metadata management unit is configured to perform the following steps:
generating seven metadata objects, wherein the seven metadata objects comprise an object class, a property, a classification scheme, a data element concept, a value range, a data element and a data set specification, the object class and the property form the data element concept, the value range is formulated according to the classification scheme, the data element is formed according to the data element concept and the value range, and a plurality of data elements form the data set specification;
constructing a base component of the metadata object, wherein the base component comprises multiple types of attributes, and the multiple types of attributes comprise names, definitions, constraints, conditions, value ranges, occurrence times, data types, maximum lengths and remarks.
Further, the normalization processing unit is configured to perform the following steps:
carrying out standardized processing on the ecological environment data through data definition, data classification, recording format, conversion rule and coding rule;
establishing dynamically definable and maintainable coding rules and data organization standards for the standardized data;
and uniformly organizing and comprehensively managing different system data and different types of pollution source data according to the dynamically definable and maintainable coding rules and the data organization standards.
Further, the data storage management unit is configured to perform the steps of:
converting uniformly organized and comprehensively managed data through metadata information and an ETL tool, wherein the metadata information comprises database table information, data source connection information and data field authority;
dividing the converted data into batch data and stream data through the ETL tool;
respectively carrying out batch processing and stream processing on the batch data and the stream data;
and storing the processed data into a data storage area.
Further, the metadata area management unit is configured to perform the steps of:
converting uniformly organized and comprehensively managed data through metadata information and an ETL tool, wherein the metadata information comprises database table information, data source connection information and data field authority;
dividing the converted data into batch data and stream data through the ETL tool;
respectively carrying out batch processing and stream processing on the batch data and the stream data;
and storing the processed data in a metadata area.
Further, the service data area unit is configured to perform the following steps:
converting uniformly organized and comprehensively managed data through metadata information and an ETL tool, wherein the metadata information comprises database table information, data source connection information and data field authority;
dividing the converted data into batch data and stream data through the ETL tool;
respectively carrying out batch processing and stream processing on the batch data and the stream data;
and storing the processed data in a service data area.
Further, the external data service module comprises a data application metadata management unit and a data application adapter management unit;
the data application metadata management unit is used for executing the following steps:
converting uniformly organized and comprehensively managed data through metadata information and an ETL tool, wherein the metadata information comprises database table information, data source connection information and data field authority;
dividing the converted data into batch data and stream data through the ETL tool;
respectively carrying out batch processing and stream processing on the batch data and the stream data;
storing the processed data in a data application metadata area;
the data application adapter management unit is used for executing the following steps:
collecting external data through a button;
and mapping the field names of the external data and the table fields in the dynamic activity level library of the ecological environment data one by one.
In yet another aspect, the present application further provides an ecological environment data dynamic activity level library terminal, including the ecological environment data dynamic activity level library as described above.
The technical scheme of the application has the following beneficial effects:
the ecological environment data dynamic activity level library acquires ecological environment data through the data acquisition and input module, achieves cleaning of the ecological environment data according to the data standard management module to obtain a complete data set, and shares the cleaned complete data set. The method and the system can integrate scattered, disordered and non-uniform ecological environment data, provide a data source with complete and standard data for an external system needing to use the ecological environment data, and improve the utilization rate of the ecological environment data.
Drawings
In order to more clearly explain the technical solution of the present application, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious to those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic diagram of a dynamic activity level library of ecological environment data provided in an embodiment of the present application.
Detailed Description
In order to make the technical solutions in the embodiments of the present application better understood and make the above objects, features and advantages of the embodiments of the present application more comprehensible, the technical solutions in the embodiments of the present application are described in further detail below with reference to the accompanying drawings.
Referring to fig. 1, a schematic diagram of a dynamic activity level library of ecological environment data provided in an embodiment of the present application is shown.
The application provides an ecological environment data dynamic activity level library, which comprises a data acquisition and input module, a data standard management module and an external data service module.
The data acquisition and input module comprises a data acquisition unit, an acquisition and input metadata management unit, an adapter management unit and a message bus management unit.
The data acquisition unit is used for acquiring ecological environment data; the ecological environment data is data which is obtained from an existing ecological environment government function department and an information-based system under construction, and different architecture systems such as a pollution source online monitoring platform, a pollution discharge license system, a secondary pollution system (a national secondary pollution source general survey system), an enterprise electricity utilization monitoring system, intelligent environment-friendly series application and the like are existing and established in the ecological environment government function department. The ecological environment data dynamic level library is designed by taking an ETL theory and a Stream processing theory as a core, and provides standardized, normalized and specialized data support for different architecture systems.
And (2) a sewage purification system: the nationwide secondary pollution source census system is used for counting pollution factors and emission indexes in nationwide pollution sources.
ETL tool: Extract-Transform-Load, which is a generic name of tools for extracting, converting and loading data to a destination, common ETL tools include key, datax, streammeters, and the like.
Wherein, button: an ETL tool is used for extracting, converting and loading data to a destination end; quartz: a task scheduling framework for periodic execution of ETL tasks; a flume: the tool is a highly available, highly reliable and distributed tool for collecting, aggregating and transmitting mass logs provided by Cloudera; kafka: a high throughput distributed publish-subscribe messaging system that can handle all the activity flow data of a consumer in a website; elastic search: a distributed multi-user capable full text search engine.
The collection entry metadata management unit is used for constructing metadata objects and basic components of the metadata objects. The following steps are specifically executed:
generating seven metadata objects by referring to the standard specification of 'environmental information metadata specification HJ 720-2017', wherein the seven metadata objects comprise object classes, characteristics, a classification scheme, a data element concept, a value range, data elements and a data set specification, the object classes and the characteristics form the data element concept, the value range is formulated according to the classification scheme, the data elements are formed according to the data element concept and the value range, and a plurality of data elements form the data set specification;
and constructing a basic component of the metadata object, wherein the basic component comprises multiple types of attributes, and the multiple types of attributes comprise names, definitions, constraints, conditions, value ranges, occurrence times, data types, maximum lengths and remarks. Wherein, constraint design is carried out according to a third model; and performing primary key dependency constraint by using the primary key ID, and realizing the n: m association relationship by using an intermediate mapping table.
The adapter management unit is used for mapping an application data structure of an external system into a data structure of an ODS raw data area. The ecological environment data acquisition is based on data existing in ecological environment government functional departments and in information-based systems under construction, and classification adaptation is carried out on structured data, semi-structured data and unstructured data to form two data ETL processing engines which are respectively a batch processing engine and a stream processing engine. The batch processing engine is based on a button and a quartz, and mainly aims at periodic, non-real-time and batch-separable data, including library-to-library migration, manual entry, file template import and the like; the stream processing engine is based on flash, kafka and elastic search and mainly aims at real-time and streaming data.
The message bus management unit is used for subscribing key nodes generated in the data acquisition process, and the key nodes are generated during data collection, whole data processing and data persistence. The method adopts a pubsub communication mode, and carries out one-to-many real-time sharing on data through a topic message publishing and subscribing model, and the whole message bus management unit consists of three participants which are respectively a producer providing data, a consumer needing data and a message agent. The message agent uses an Actor programming model to realize decentralized design and meet the flexible capacity expansion requirement of the service level. And the producer and the consumer realize the sending and receiving of the message through the client SDK.
The data standard management module comprises a standardization processing unit, an ODS original data area management unit, a data storage area management unit, a metadata area management unit and a service data area management unit.
The standardized processing unit is used for carrying out standardized processing on the ecological environment data and carrying out unified organization and comprehensive management on different data in the ecological environment data. The standardization processing unit is used for executing the following steps:
carrying out standardized processing on the ecological environment data through data definition, data classification, recording format, conversion rule and coding rule; the data definition, data classification, recording format and conversion/coding rule can be set according to the national standard;
establishing dynamically definable and maintainable coding rules and data organization standards for the standardized data;
and uniformly organizing and comprehensively managing different system data and different types of pollution source data according to dynamically defined and maintainable coding rules and data organization standards, so as to realize cleaning of ecological environment data. For example: collected SO2Is not within the standard range set by the data organization standard, the SO is added2The value of (2) is identified as erroneous data and the data is purged.
And the ODS original data area management unit is used for storing the data acquired by the data acquisition and entry module into the ODS original data area.
The data storage area management unit is used for converting and classifying the data which are uniformly organized and comprehensively managed and storing the converted and classified data into the data storage area; the data storage area management unit is used for executing the following steps:
converting uniformly organized and comprehensively managed data through metadata information and an ETL tool, wherein the metadata information comprises database table information, data source connection information and data field authority;
dividing the converted data into batch data and stream data through an ETL tool;
respectively carrying out batch processing and stream processing on batch data and stream data, wherein the batch processing is triggered by a timing task or manually, and the stream processing is triggered when an ETL (extract transform load) task is started to be collected;
and storing the processed data into a data storage area. The data storage area is provided with a plurality of data set lists, such as an environmental protection knowledge base, a grid data set and the like. And storing the processed data into a corresponding data set list according to the properties. Such as policies, guidelines or science popularization knowledge issued by the environmental protection department, etc. are stored in the environmental protection knowledge base; the geographical location of the pollution source is stored in a grid dataset.
An environment-friendly knowledge base: and the data warehouse related to the environmental protection is used for storing the related content of the environmental protection information. For example, relevant documents such as 'urban environmental air quality ranking technical regulation' and 'environmental air quality standard' issued by the environmental protection department are stored in an environmental protection knowledge base.
Grid data: and the designated area is divided into grids, the area of each cell is responsible for one grid member, and the grid member is responsible for the pollution source statistics of the area and processes tasks issued by related departments. The grid data comprises basic information of grid members, information of relevant cases processed by the grid members, information of pollution sources counted by the grid members and the like.
The metadata area management unit is used for converting and classifying the data which are uniformly organized and comprehensively managed, and storing the converted and classified data into the metadata area. The metadata area management unit is configured to perform the steps of:
converting uniformly organized and comprehensively managed data through metadata information and an ETL tool, wherein the metadata information comprises database table information, data source connection information and data field authority;
dividing the converted data into batch data and stream data through an ETL tool;
respectively carrying out batch processing and stream processing on batch data and stream data, wherein the batch processing is triggered by a timing task or manually, and the stream processing is triggered when an ETL (extract transform load) task is started to be collected;
and storing the processed data in a metadata area. The metadata area has a plurality of data set lists, such as a data quality analysis list, a data change record list and a data compliance audit list. And storing the processed data in a data set list corresponding to the band according to the properties.
The data quality analysis table is mainly used for comparing data after data cleaning with data before cleaning, and counting and analyzing which data belong to dirty data and which data are available data in the cleaning process. The smaller the change in the data amount before and after data cleansing, the higher the data quality.
The data change recording list is used for recording the process of changing each data index in the data source.
The data compliance audit list is used for carrying out statistical analysis on whether the acquired data meet the data and standard specifications or not and recording data indexes meeting the data and standard specifications.
The service data area management unit is used for converting and classifying the data which are uniformly organized and comprehensively managed, and storing the converted and classified data into the service data area. The service data area unit is used for executing the following steps:
converting uniformly organized and comprehensively managed data through metadata information and an ETL tool, wherein the metadata information comprises database table information, data source connection information and data field authority;
dividing the converted data into batch data and stream data through an ETL tool;
respectively carrying out batch processing and stream processing on batch data and stream data, wherein the batch processing is triggered by a timing task or manually, and the stream processing is triggered when an ETL (extract transform load) task is started to be collected;
and storing the processed data in a service data area. The service data area has a plurality of data set lists, such as data sets and indexes.
The data set refers to a data set which is formed by cleaning, filtering and processing the acquired data according to a data standard specification and can serve other systems; the index refers to a certain data in the collected data. Such as SO in the atmosphere2Concentration is an indicator in the data.
The external data service module is used for checking and extracting data of the service data area, and the data of the service data area is corresponding data generated according to the service scene. The external data service module includes a data application metadata management unit and a data application adapter management unit.
The data application metadata management unit is used for executing the following steps:
converting uniformly organized and comprehensively managed data through metadata information and an ETL tool, wherein the metadata information comprises database table information, data source connection information and data field authority;
dividing the converted data into batch data and stream data through an ETL tool;
respectively carrying out batch processing and stream processing on batch data and stream data;
and storing the processed data in a data application metadata area.
The data application adapter management unit is used for executing the following steps:
collecting external data through a button;
and mapping the field name of the external data and the table field in the dynamic library in the dynamic activity level library of the ecological environment data one by one to ensure that the field name of each external data source can correspond to the field name of one table in the dynamic library.
Furthermore, the data of the service data area is used as a database for an external system requiring ecological environment data to extract, so that a reliable data source is provided for the external system requiring ecological environment data, and the purpose of sharing the ecological environment data is achieved. The external system comprises a pollution source supervision intelligent and data fusion analysis system, a pollution source wall map combat system, an air quality control special class auxiliary system, a pollution source spectrum system penetrating system, an ecological environment scheduling command system, a heavy pollution emergency control system and the like. For example, when the service scene is a pollution source supervision service, the data such as the geographical position of an enterprise, the information of the enterprise and the industry and the business, the pollution discharge license of the enterprise, the water pollution level and the like of the enterprise with water pollution can be acquired from the service data area; the data of the service data area can be checked by data users having authority to access the database, such as supervision departments, environmental protection departments and the like, through the external data service module, and the data of the service data area can also be extracted or exported, and the required data is collated and analyzed to generate supervision data, so that a basis is provided for the supervision departments to enforce law.
The ecological environment data dynamic activity level library acquires ecological environment data through the data acquisition and input module, achieves cleaning of the ecological environment data according to the data standard management module to obtain a complete data set, and shares the cleaned complete data set. The method and the system can integrate scattered, disordered and non-uniform ecological environment data, provide a database with complete and standard data for an external system needing to use the ecological environment data, and improve the utilization rate of the ecological environment data.
The application also provides a dynamic activity level library terminal of the ecological environment data, which comprises the dynamic activity level library of the ecological environment data.
The embodiments provided in the present application are only a few examples of the general concept of the present application, and do not limit the scope of the present application. Any other embodiments extended according to the scheme of the present application without inventive efforts will be within the scope of protection of the present application for a person skilled in the art.
Claims (8)
1. The system comprises an ecological environment data dynamic activity level library, a data management module and an external data service module, wherein the level library comprises a data acquisition and input module, a data standard management module and an external data service module; it is characterized in that the preparation method is characterized in that,
the data acquisition and input module comprises a data acquisition unit, an acquisition and input metadata management unit, an adapter management unit and a message bus management unit; the data acquisition unit is used for acquiring ecological environment data; the collection and entry metadata management unit is used for constructing metadata objects and basic components of the metadata objects; the adapter management unit is used for mapping an application data structure of an external system to a data structure of an ODS original data area; the message bus management unit is used for subscribing key nodes generated in the data acquisition process, wherein the key nodes are generated during data collection, whole data processing and data persistence;
the data standard management module comprises a standardized processing unit, an ODS original data area management unit, a data storage area management unit, a metadata area management unit and a service data area management unit; the standardized processing unit is used for carrying out standardized processing on the ecological environment data and carrying out unified organization and comprehensive management on different data in the ecological environment data; the ODS original data area management unit is used for storing the data acquired by the data acquisition and input module into the ODS original data area; the data storage area management unit is used for converting and classifying the data which are uniformly organized and comprehensively managed, and storing the converted and classified data into the data storage area; the metadata area management unit is used for converting and classifying the data which are uniformly organized and comprehensively managed, and storing the converted and classified data into a metadata area; the business data area management unit is used for converting and classifying the data which are uniformly organized and comprehensively managed, and storing the converted and classified data into the business data area;
the external data service module is used for checking and extracting the data of the service data area, wherein the data of the service data area is corresponding data generated according to a service scene.
2. The dynamic activity level repository for ecological environment data according to claim 1, wherein the collection entry metadata management unit is configured to perform the following steps:
generating seven metadata objects, wherein the seven metadata objects comprise an object class, a property, a classification scheme, a data element concept, a value range, a data element and a data set specification, the object class and the property form the data element concept, the value range is formulated according to the classification scheme, the data element is formed according to the data element concept and the value range, and a plurality of data elements form the data set specification;
constructing a base component of the metadata object, wherein the base component comprises multiple types of attributes, and the multiple types of attributes comprise names, definitions, constraints, conditions, value ranges, occurrence times, data types, maximum lengths and remarks.
3. The dynamic activity level library of ecological environment data according to claim 1, wherein the normalization processing unit is configured to perform the following steps:
carrying out standardized processing on the ecological environment data through data definition, data classification, recording format, conversion rule and coding rule;
establishing dynamically definable and maintainable coding rules and data organization standards for the standardized data;
and uniformly organizing and comprehensively managing different system data and different types of pollution source data according to the dynamically definable and maintainable coding rules and the data organization standards.
4. The dynamic activity level library of ecosystem data according to claim 3, wherein the data storage management unit is configured to perform the following steps:
converting uniformly organized and comprehensively managed data through metadata information and an ETL tool, wherein the metadata information comprises database table information, data source connection information and data field authority;
dividing the converted data into batch data and stream data through the ETL tool;
respectively carrying out batch processing and stream processing on the batch data and the stream data;
and storing the processed data into a data storage area.
5. The dynamic activity level library of ecosystem data according to claim 3, wherein the metadata region management unit is configured to perform the following steps:
converting uniformly organized and comprehensively managed data through metadata information and an ETL tool, wherein the metadata information comprises database table information, data source connection information and data field authority;
dividing the converted data into batch data and stream data through the ETL tool;
respectively carrying out batch processing and stream processing on the batch data and the stream data;
and storing the processed data in a metadata area.
6. The dynamic activity level library of ecological environment data according to claim 3, wherein the service data area unit is configured to perform the following steps:
converting uniformly organized and comprehensively managed data through metadata information and an ETL tool, wherein the metadata information comprises database table information, data source connection information and data field authority;
dividing the converted data into batch data and stream data through the ETL tool;
respectively carrying out batch processing and stream processing on the batch data and the stream data;
and storing the processed data in a service data area.
7. The dynamic activity level library of ecosystem data according to claim 3, wherein the external data service module comprises a data application metadata management unit and a data application adapter management unit;
the data application metadata management unit is used for executing the following steps:
converting uniformly organized and comprehensively managed data through metadata information and an ETL tool, wherein the metadata information comprises database table information, data source connection information and data field authority;
dividing the converted data into batch data and stream data through the ETL tool;
respectively carrying out batch processing and stream processing on the batch data and the stream data;
storing the processed data in a data application metadata area;
the data application adapter management unit is used for executing the following steps:
collecting external data through a button;
and mapping the field names of the external data and the table fields in the dynamic activity level library of the ecological environment data one by one.
8. Ecological environment data dynamic activity level library terminal, characterized in, that it comprises an ecological environment data dynamic activity level library according to any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111502124.6A CN113901042A (en) | 2021-12-10 | 2021-12-10 | Ecological environment data dynamic activity level library and terminal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111502124.6A CN113901042A (en) | 2021-12-10 | 2021-12-10 | Ecological environment data dynamic activity level library and terminal |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113901042A true CN113901042A (en) | 2022-01-07 |
Family
ID=79025705
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111502124.6A Pending CN113901042A (en) | 2021-12-10 | 2021-12-10 | Ecological environment data dynamic activity level library and terminal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113901042A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116304258A (en) * | 2023-05-15 | 2023-06-23 | 上海爱可生信息技术股份有限公司 | Retrieval method, retrieval system and readable storage medium based on vector database |
| CN116383222A (en) * | 2023-03-28 | 2023-07-04 | 中国标准化研究院 | Construction method of grass ecological restoration monitoring core database |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107992552A (en) * | 2017-11-28 | 2018-05-04 | 南京莱斯信息技术股份有限公司 | A kind of data interchange platform and method for interchanging data |
| CN110502559A (en) * | 2019-07-25 | 2019-11-26 | 浙江公共安全技术研究院有限公司 | A kind of data/address bus and transmission method of credible and secure cross-domain data exchange |
| CN111198918A (en) * | 2020-01-17 | 2020-05-26 | 国网福建省电力有限公司 | Data processing system based on big data platform and link optimization method |
| CN111459908A (en) * | 2020-03-08 | 2020-07-28 | 中国科学院城市环境研究所 | Multi-source heterogeneous ecological environment big data processing method and system based on data lake |
| CN112163724A (en) * | 2020-08-05 | 2021-01-01 | 宁夏无线互通信息技术有限公司 | Environment information data resource integration system |
| CN112231333A (en) * | 2020-11-09 | 2021-01-15 | 南京莱斯网信技术研究院有限公司 | Ecological environment data sharing and exchanging method and system |
| CN112699175A (en) * | 2021-01-15 | 2021-04-23 | 广州汇智通信技术有限公司 | Data management system and method thereof |
| CN112926852A (en) * | 2021-02-19 | 2021-06-08 | 南京莱斯网信技术研究院有限公司 | Atmospheric ecological environment analysis method based on data fusion |
| CN112965948A (en) * | 2020-12-30 | 2021-06-15 | 中科环森智慧科技(苏州)有限公司 | Management service center system based on data |
-
2021
- 2021-12-10 CN CN202111502124.6A patent/CN113901042A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107992552A (en) * | 2017-11-28 | 2018-05-04 | 南京莱斯信息技术股份有限公司 | A kind of data interchange platform and method for interchanging data |
| CN110502559A (en) * | 2019-07-25 | 2019-11-26 | 浙江公共安全技术研究院有限公司 | A kind of data/address bus and transmission method of credible and secure cross-domain data exchange |
| CN111198918A (en) * | 2020-01-17 | 2020-05-26 | 国网福建省电力有限公司 | Data processing system based on big data platform and link optimization method |
| CN111459908A (en) * | 2020-03-08 | 2020-07-28 | 中国科学院城市环境研究所 | Multi-source heterogeneous ecological environment big data processing method and system based on data lake |
| CN112163724A (en) * | 2020-08-05 | 2021-01-01 | 宁夏无线互通信息技术有限公司 | Environment information data resource integration system |
| CN112231333A (en) * | 2020-11-09 | 2021-01-15 | 南京莱斯网信技术研究院有限公司 | Ecological environment data sharing and exchanging method and system |
| CN112965948A (en) * | 2020-12-30 | 2021-06-15 | 中科环森智慧科技(苏州)有限公司 | Management service center system based on data |
| CN112699175A (en) * | 2021-01-15 | 2021-04-23 | 广州汇智通信技术有限公司 | Data management system and method thereof |
| CN112926852A (en) * | 2021-02-19 | 2021-06-08 | 南京莱斯网信技术研究院有限公司 | Atmospheric ecological environment analysis method based on data fusion |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116383222A (en) * | 2023-03-28 | 2023-07-04 | 中国标准化研究院 | Construction method of grass ecological restoration monitoring core database |
| CN116304258A (en) * | 2023-05-15 | 2023-06-23 | 上海爱可生信息技术股份有限公司 | Retrieval method, retrieval system and readable storage medium based on vector database |
| CN116304258B (en) * | 2023-05-15 | 2023-07-21 | 上海爱可生信息技术股份有限公司 | Retrieval method, retrieval system and readable storage medium based on vector database |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109684352B (en) | Data analysis system, data analysis method, storage medium, and electronic device | |
| CN112163724A (en) | Environment information data resource integration system | |
| CN108039959B (en) | Data situation perception method, system and related device | |
| CN113901042A (en) | Ecological environment data dynamic activity level library and terminal | |
| CN107451149B (en) | Monitoring method and device for flow data query task | |
| CN111459908A (en) | Multi-source heterogeneous ecological environment big data processing method and system based on data lake | |
| CN107958080A (en) | A kind of big data report processing method based on ElasticSearch | |
| CN111046059B (en) | Low-efficiency SQL statement analysis method and system based on distributed database cluster | |
| CN111400288A (en) | Data quality inspection method and system | |
| US20200334314A1 (en) | Emergency disposal support system | |
| CN108268569A (en) | The acquisition of water resource monitoring data and analysis system and method based on big data technology | |
| CN114218218A (en) | Data processing method, device and equipment based on data warehouse and storage medium | |
| CN104616100A (en) | Management system for emergency plan | |
| CN115827797A (en) | Environmental data analysis and integration method and system based on big data | |
| CN103902582A (en) | Data warehouse redundancy reduction method and device | |
| CN114265883B (en) | Method, equipment and storage medium for real-time data management | |
| CN117251414B (en) | Data storage and processing method based on heterogeneous technology | |
| CN117217412B (en) | Waste free urban construction management system based on resource utilization | |
| CN110826845B (en) | Multidimensional combination cost allocation device and method | |
| CN116010380A (en) | Data warehouse automatic management method based on visual modeling | |
| CN106503118B (en) | A kind of data acquisition system and its implementation based on HC-TABLE | |
| CN114357082A (en) | Cloud computing-based big data analysis method and system | |
| CN112540843B (en) | Resource allocation method and device, storage equipment and storage medium | |
| CN113157924A (en) | Smart city-oriented resident public demand response method | |
| CN112395291A (en) | Method and system for dynamically generating wide table according to data assets |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20220107 |