CN117473024A - Water conservancy industry data management resource pool design method based on big data platform - Google Patents
Water conservancy industry data management resource pool design method based on big data platform Download PDFInfo
- Publication number
- CN117473024A CN117473024A CN202311547828.4A CN202311547828A CN117473024A CN 117473024 A CN117473024 A CN 117473024A CN 202311547828 A CN202311547828 A CN 202311547828A CN 117473024 A CN117473024 A CN 117473024A
- Authority
- CN
- China
- Prior art keywords
- data
- water conservancy
- resource pool
- platform
- industry
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 title claims abstract description 270
- 238000013523 data management Methods 0.000 title claims abstract description 86
- 238000013461 design Methods 0.000 title claims abstract description 65
- 238000000034 method Methods 0.000 title claims abstract description 62
- 230000008569 process Effects 0.000 claims abstract description 14
- 238000012545 processing Methods 0.000 claims description 26
- 238000011161 development Methods 0.000 claims description 25
- 238000011835 investigation Methods 0.000 claims description 22
- 239000011159 matrix material Substances 0.000 claims description 20
- 238000004140 cleaning Methods 0.000 claims description 18
- 238000004458 analytical method Methods 0.000 claims description 17
- 238000012544 monitoring process Methods 0.000 claims description 12
- 230000006870 function Effects 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 10
- 238000009960 carding Methods 0.000 claims description 9
- 238000004364 calculation method Methods 0.000 claims description 7
- 238000000638 solvent extraction Methods 0.000 claims description 6
- 238000004519 manufacturing process Methods 0.000 claims description 5
- 230000004927 fusion Effects 0.000 claims description 4
- 230000008521 reorganization Effects 0.000 claims description 3
- 238000007726 management method Methods 0.000 abstract description 8
- 238000003860 storage Methods 0.000 description 19
- 238000013499 data model Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000003491 array Methods 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 4
- 230000015572 biosynthetic process Effects 0.000 description 4
- 238000013480 data collection Methods 0.000 description 4
- 238000012821 model calculation Methods 0.000 description 4
- 239000007787 solid Substances 0.000 description 4
- 238000009472 formulation Methods 0.000 description 3
- 239000000203 mixture Substances 0.000 description 3
- 241000565356 Fraxinus pennsylvanica Species 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 2
- 239000008186 active pharmaceutical agent Substances 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000013079 data visualisation Methods 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000013439 planning Methods 0.000 description 2
- 238000007639 printing Methods 0.000 description 2
- 230000001902 propagating effect Effects 0.000 description 2
- 238000005086 pumping Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 238000012800 visualization Methods 0.000 description 2
- RYGMFSIKBFXOCR-UHFFFAOYSA-N Copper Chemical compound [Cu] RYGMFSIKBFXOCR-UHFFFAOYSA-N 0.000 description 1
- 206010063385 Intellectualisation Diseases 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000009430 construction management Methods 0.000 description 1
- 229910052802 copper Inorganic materials 0.000 description 1
- 239000010949 copper Substances 0.000 description 1
- 230000007123 defense Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2465—Query processing support for facilitating data mining operations in structured databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Quality & Reliability (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The application provides a water conservancy industry data management resource pool design method based on a big data platform, which comprises the following steps: based on a water conservancy service system, acquiring an association relation and a data flow direction between the systems; establishing a water conservancy data standard specification based on national standards and industry standards; designing a data resource pool based on the water conservancy data standard specification; and developing a big data platform based on the data resource pool so as to realize the fine treatment of water conservancy data. The method and the device can improve the data management capability in the water conservancy data management process; the data value is further mined, the data utilization rate is improved, the stability of the data management model is high, the problem of inconsistent sources of water conservancy data in a complex information system environment can be solved, one source is realized, and the fine management of the water conservancy data is realized.
Description
Technical Field
The invention belongs to the technical field of hydraulic engineering, relates to a technical method for data management, and particularly relates to a design method for a data management resource pool in the hydraulic industry based on a big data platform.
Background
At present, the water conservancy industry is gradually improving informatization and intellectualization processes, utilizing new generation information technologies such as big data, artificial intelligence, internet of things, 5G, remote sensing, edge calculation, mobile interconnection, unmanned aerial vehicle and the like, focusing on water and drought disaster defense, water resource development and utilization, three red line management and control of water resources, monitoring and control of river and lake long-acting caterpillar, construction management of kaleidoscope, full life cycle management of major water conservancy engineering construction and other water conservancy key business scenes, constructing a water conservancy intelligent and efficient decision support service, and achieving the overall goal of 'retrieval anywhere, analysis anytime and accompanying command'. The data application requirements of digital twin water conservancy are met through data management, the data value is further mined, and support is provided for decision making of digital management of water conservancy.
However, in the existing data management technology in the water conservancy industry, the water conservancy data management capability is insufficient, the problems of one source, inconsistent data types and formats cannot be solved, the problem of inconsistent water conservancy data sources in the complex information system environment cannot be solved, and the definition degree of the water conservancy data management is insufficient.
Disclosure of Invention
In view of the above-mentioned drawbacks of the prior art, an object of the present application is to provide a method for designing a data management resource pool in the water conservancy industry based on a big data platform, which is used for solving the problems that in the prior art, due to insufficient water conservancy data management capability, a plurality of sources cannot be realized, the data types and formats are inconsistent, the problem that the water conservancy data sources are inconsistent in the complex information system environment cannot be solved, so that the plurality of sources cannot be realized, the refinement degree of the water conservancy data management is insufficient, and further the problems that the existing water conservancy industry data management process has insufficient data management capability, inconsistent data types or formats, excessively rough water conservancy data management is generated due to inconsistent data sources, data value cannot be further mined, the data utilization rate is low, the stability of a data management model is poor, and the like.
To achieve the above and other related objects, in a first aspect, the present application provides a method for designing a water conservancy industry data management resource pool based on a big data platform, including the following steps: based on a water conservancy service system, acquiring an association relation and a data flow direction between the systems; establishing a water conservancy data standard specification based on national standards and industry standards; designing a data resource pool based on the water conservancy data standard specification; and developing a big data platform based on the data resource pool so as to realize the fine treatment of water conservancy data.
In an implementation manner of the first aspect, based on the water conservancy service system, acquiring the association relationship and the data flow direction between the systems includes the following steps: establishing an investigation group to carry out information system investigation and acquire water conservancy system data information from a data layer; establishing a business data matrix based on the water conservancy system data information; and adopting a data exploration tool of a large data platform to automatically scan and inventory.
In one implementation manner of the first aspect, the investigation form includes: any one or more combinations of walks, questionnaires, interviews; the water conservancy system data information comprises: any one or more of water conservancy system functions, water conservancy data production and utilization status, water conservancy data types and water conservancy data flow directions.
In an implementation manner of the first aspect, the data resource pool adopts a data warehouse four-layer design structure, and the structure includes: ODS pastes source layer, DWD detail layer, DWS theme layer and ADS thematic layer; the ODS is attached to the source layer and is used for dividing data fields of water conservancy data; the DWD detail layer is used for carrying out data processing on the divided water conservancy data to obtain standardized water conservancy data; the DWS topic layer is used for fusing the standardized water conservancy data to obtain a service data table of a topic domain; the ADS thematic layer is used for carrying out water conservancy data application analysis and reorganization and library building for different professions according to user requirements so as to design various application themes and generate hydrological models in different application scenes.
In one implementation manner of the first aspect, the ODS patch source layer includes: and checking, data carding and data sorting of the water conservancy data system.
In one implementation manner of the first aspect, the data of the DWD detail layer is derived from the ODS patch source layer; the DWD detail layer includes: carrying out processes of data cleaning, data domain division, data fusion and specification on the water conservancy data according to the standards and rules; the data domain partitioning includes: any one or more combination of basic data, space remote sensing information, real-time water and rain condition monitoring, weather forecast and business data; the data cleaning is used for providing specific and definite data requirements and reference standard specifications by a user, determining cleaning rules, feeding back to the data center station and the data source, and improving the data quality.
In one implementation manner of the first aspect, the DWS theme layer includes the following steps: based on DWD detail layer data and user requirements, integrating and analyzing a water conservancy service data table of a certain topic domain in the water profit industry; according to different application scenes, arranging theme objects in each application scene, and establishing a logic model of the theme objects and a water conservancy theme object relation table to obtain association relations of the water conservancy theme objects; acquiring water conservancy holographic data under a certain application scene through the association relation between the theme object and the water conservancy theme object; wherein the logic model comprises: any one or more of water conservancy theme object foundation, real-time water rain condition, weather and space dimension information are combined; the water conservancy theme object comprises: any one or more of a river basin, river, lake, reservoir dam, hydrological station, sluice, pump station.
In an implementation manner of the first aspect, the ADS thematic layer is designed for input data obtained by calculation of a hydrological model; the hydrological model comprises: a forecasting model, a one-dimensional evolution model, a two-dimensional evolution model and an early warning model.
In a second aspect, the present application provides a water conservancy industry data management resource pool design system based on a big data platform, including: the acquisition module is used for acquiring the association relation and the data flow direction among the systems based on the water conservancy service system; the standard making module is used for making water conservancy data standard specifications based on national standards and industry standards; the design module is used for designing a data resource pool based on the water conservancy data standard specification; and the platform development module is used for carrying out large data platform development based on the data resource pool so as to realize the fine treatment of water conservancy data.
In a final aspect, the present application provides a water conservancy industry data management resource pool design device based on big data platform, including: a processor and a memory. The memory is used for storing a computer program; the processor is connected with the memory and is used for executing the computer program stored in the memory so that the water conservancy industry data management resource pool design device based on the big data platform executes the water conservancy industry data management resource pool design method based on the big data platform.
As described above, the water conservancy industry data management resource pool design method based on the big data platform has the following beneficial effects:
the application provides a design method of a water conservancy industry data management resource pool based on a big data platform, which realizes cross-business and cross-organization integrated data service and linkage by establishing a digital twin water conservancy data resource pool, provides standard authoritative data for service matters and application matters continuously changed in the digital twin engineering, and finally provides a solid foundation for building an authoritative data center. Meanwhile, the data management capability in the water conservancy data management process can be improved; the data value is further mined, the data utilization rate is improved, the stability of the data management model is high, the problem of inconsistent sources of water conservancy data in a complex information system environment can be solved, one source is realized, and the fine management of the water conservancy data is realized.
Drawings
FIG. 1 is a flow chart of a method for designing a water conservancy industry data management resource pool based on a big data platform according to an embodiment of the present invention.
Fig. 2 is a schematic flow chart of S11 in the method for designing a water conservancy industry data management resource pool based on a big data platform.
Fig. 3 is a schematic flow chart of S13 in the method for designing a water conservancy industry data management resource pool based on a big data platform.
Fig. 4 shows a schematic diagram of a four-layer design structure of a data warehouse in the design method of the data management resource pool of the water conservancy industry based on a big data platform.
FIG. 5 is a schematic diagram of a hydraulic industry data management resource pool design system based on a big data platform according to an embodiment of the present invention.
FIG. 6 is a schematic diagram of a hydraulic industry data management resource pool design device based on a big data platform according to an embodiment of the present invention.
Description of element reference numerals
51. Acquisition module
52. Standard making module
53. Design module
54. Platform development module
61. Processor and method for controlling the same
62. Memory device
S11 to S14 steps
Detailed Description
Other advantages and effects of the present invention will become apparent to those skilled in the art from the following disclosure, which describes the embodiments of the present invention with reference to specific examples. The invention may be practiced or carried out in other embodiments that depart from the specific details, and the details of the present description may be modified or varied from the spirit and scope of the present invention. It should be noted that the following embodiments and features in the embodiments may be combined with each other without conflict.
It should be noted that the illustrations provided in the following embodiments merely illustrate the basic concept of the present invention by way of illustration, and only the components related to the present invention are shown in the drawings and are not drawn according to the number, shape and size of the components in actual implementation, and the form, number and proportion of the components in actual implementation may be arbitrarily changed, and the layout of the components may be more complicated.
The method for designing the water conservancy industry data management resource pool based on the big data platform provided in the embodiment of the application will be described in detail below with reference to the accompanying drawings in the embodiment of the application. The method is used for solving the problems that in the prior art, due to the fact that the water conservancy data management capability is insufficient, one source cannot be realized, the data types and formats are inconsistent, the problem that the water conservancy data sources are inconsistent under a complex information system environment cannot be solved, one source cannot be realized, so that the refinement degree of the water conservancy data management is insufficient, and further the problems that the data management capability is insufficient, the data types or formats are not uniform, the data sources are inconsistent, the water conservancy data management is too rough, the data value cannot be further mined, the data utilization rate is low, the stability of a data management model is poor and the like are caused in the existing water conservancy industry data management process.
Referring to fig. 1, a flow chart of a method for designing a water conservancy industry data management resource pool based on a big data platform according to an embodiment of the invention is shown. As shown in fig. 1, the embodiment provides a water conservancy industry data management resource pool design method based on a big data platform.
The water conservancy industry data management resource pool design method based on the big data platform specifically comprises the following steps:
s11, based on the water conservancy service system, the association relation and the data flow direction between the systems are obtained, and based on the water conservancy service system, the association relation and the data flow direction between the systems are obtained.
The robot formation in this application is fixed, but its position is uncertain. That is, the initial and arrival positions of the formation are not fixed in a Cartesian coordinate system. Thus, a determination of the coordinates is required.
And S111, carrying out information system investigation by an investigation group, and acquiring the water conservancy system data information from the data layer.
In this embodiment, an investigation group is established, and according to actual conditions, information system investigation is performed through a plurality of investigation shapes, and water conservancy system data information is obtained from a data layer. Among the various investigation forms include: different forms of interviews, questionnaires, interviews, etc.; the water conservancy system data information comprises: the hydraulic system comprises a hydraulic system function, a hydraulic data production and utilization status, a hydraulic data type, a hydraulic data flow direction and a plurality of hydraulic data information.
Specifically, the information system investigation is performed by the investigation group in the forms of interview, questionnaire, interview, etc., and the information such as system functions, data production and utilization status, data types, data flow direction, etc. is grasped from the data layer.
S112, establishing a business data matrix based on the water conservancy system data information.
In this embodiment, a service data matrix is established to provide a basis for designing a data resource pool subsequently.
Specifically, according to the service requirements and objectives, it is determined from which data sources data needs to be acquired, for example: databases, reports, APIs, etc. Formulating a data collection plan based on the determined data sources, comprising: collection range, collection frequency, collection time, etc. of the data. Data is then acquired from the data sources according to the formulated data collection plan and cleaned and consolidated as necessary. According to the business requirements and the targets, a model and a structure of the data matrix are determined, for example, what data are respectively represented by rows and columns of the matrix, association relations among the data and the like. Then, according to the determined data matrix model, filling the collected data into the matrix, and carrying out necessary calculation and processing; after the data matrix is constructed, verification is needed to ensure the accuracy and the integrity of the data; once the data matrix is validated, the data matrix may be published for use by subsequent data analysis and decision support.
S113, automatically scanning and checking by adopting a data exploration tool of a large data platform.
In this embodiment, for structured data in the service system, a data exploration tool of a large data platform is used to automatically scan and inventory.
Specifically, depending on the traffic requirements and the size of the data volume, a suitable data exploration tool is selected, for example: tableau, powerBI, redash, etc. The selected data exploration tool is then connected to the large data platform to enable storage and analysis of the data. A view of the data is created on a large data platform to more conveniently view and analyze the data. And then using an automatic scanning function in the data exploration tool to automatically scan the structured data so as to find out the abnormality and the problem in the data. And using a data checking function in the data exploration tool to check the structured data so as to know the distribution, trend and association relation of the data. Then, according to the scanning and checking results, the problems and opportunities in the data are analyzed, and corresponding suggestions and measures are provided. Finally, the analysis result is exported to a designated storage device or shared folder for subsequent use.
Through the steps, the structured data in the business system can be automatically scanned and checked by using the data exploration tool of the big data platform, so that enterprises can be helped to better know and utilize the data resources.
And S12, formulating water conservancy data standard specifications based on national standards and industry standards.
In the embodiment, a water conservancy data standard specification is determined, and national standards and industry specifications of water conservancy industry are collected; then uniformly making water conservancy data standard; respectively formulating a data sharing specification, a data management specification and an application development specification; and writing a standard specification document, and then issuing a popularization standard specification. In addition, attention is paid to updating of the standard specification.
Specifically, the objectives of the water conservancy data standard specification are specified, for example: standardizing data formats, improving data quality, facilitating data sharing, etc., and determining the applicable scope of standard specifications, for example: applicable to areas, applicable to business fields, etc. Then, the national standard and industry standard related to the water conservancy data, and related technical literature and data are collected for analysis and research. And then, according to the collected data and analysis results, formulating water conservancy data standards including data formats, data types, data precision, data element definitions and the like. Then, formulating a water conservancy data sharing specification, including: sharing mode, sharing flow, sharing protocol, sharing security, etc.; formulating a water conservancy data management specification, comprising: data acquisition, data processing, data storage, data backup and the like; formulating a water conservancy application development specification, comprising: application development flow, application interface design, etc. Next, the established water conservancy data standard specification is written into a document, which comprises the following steps: standard numbers, names, version numbers, release dates, etc., and typesetting and printing according to the document format. Finally, release and popularization standard specifications: publishing and popularizing the written water conservancy data standard specification document, which comprises the following steps: release to related departments of water conservancy industry, popularization to related enterprises and institutions, and the like.
Note that the standard specification is maintained and updated continuously: and continuously maintaining and updating the formulated water conservancy data standard specification according to the actual application condition and the feedback opinion so as to ensure the adaptability and the effectiveness of the water conservancy data standard specification.
And S13, designing a data resource pool based on the water conservancy data standard specification. Referring to fig. 3, a flow chart of S13 in the method for designing a water conservancy industry data management resource pool based on a big data platform according to the present invention is shown. As shown in fig. 3, the step S13 includes the following steps:
the data resource pool adopts a typical four-layer design of a data warehouse and is divided into: ODS pastes source layer, DWD detail layer, DWS theme layer, ADS thematic layer.
S131, the ODS is attached to the source layer and used for dividing data fields of water conservancy data.
In this embodiment, the ODS paste source layer includes: and checking, data carding and data sorting of the water conservancy data system.
Specifically, the layer basically does not change the original data, so as to solve the problem of possible data tracing. And dividing the data domain and referring to the data source.
The layer is accessed to all original data of the existing system and the system to be built, and the accessed data is basically not changed and processed, so that the problem of tracing the data source is prevented. The data of the layer corresponds to the data source system. The working focus of the source layer is on system checking and data carding, and data useful for each scale section of the project are tidied.
And S132, the DWD detail layer is used for carrying out data processing on the divided water conservancy data to obtain standardized water conservancy data.
The data is from ODS paste source layer, and is cleaned, integrated and normalized according to standard, rule and the like. Dirty data, garbage data, data with inconsistent specifications, inconsistent state definitions and inconsistent naming standards are processed to provide standard data for the DWS topic layer.
In this embodiment, the data of the DWD detail layer is derived from the ODS paste source layer; the DWD detail layer includes: carrying out processes of data cleaning, data domain division, data fusion and specification on the water conservancy data according to the standards and rules; the data domain partitioning includes: any one or more combination of basic data, space remote sensing information, real-time water and rain condition monitoring, weather forecast and business data; the data cleaning is used for providing specific and definite data requirements and reference standard specifications by a user, determining cleaning rules, feeding back to the data center station and the data source, and improving the data quality.
Specifically, the DWD layer performs cleaning, integration, and standardization according to standards, rules, and the like. The design emphasis is on data domain partitioning and the formulation of data cleansing criteria.
The data domain is divided, and the technical requirement (trial) of a provincial mountain torrent disaster monitoring, forecasting and early warning platform is referred to (9 months in 2020). The method is divided into basic data types, space remote sensing information types, real-time water and rain condition monitoring types, weather forecast types and business data types.
In addition, the data cleaning standard refers to a real-time rain condition database table structure and an identifier standard (SL 323-2011), a mountain torrent disaster investigation and evaluation database table structure and an identifier standard (2015, 7 months), a flood control project database table structure and an identifier (2015, 2 months), and a real-time rain condition database table structure and an identifier standard (SL 323-2011).
It should be noted that data cleansing is a long-term and durable operation, and in the process of system application, specific and definite data requirements and reference standard specifications are required to be proposed by users, cleansing rules are determined and fed back to the data center station and the data source to improve data quality.
The ODS original layer and the DWD detail layer are used for carding and organizing data in a bottom-up mode, and the DWS theme layer is used for monitoring mountain torrent disasters.
And S133, the DWS topic layer is used for fusing the standardized water conservancy data to obtain a service data table of a topic domain.
Based on the DWD detail layer data, service data of a certain topic domain is integrated and summarized and analyzed, typically in a broad table.
In the embodiment, a water conservancy service data table for summarizing and analyzing a certain topic domain in the water profit industry is integrated based on DWD detail layer data and user requirements; according to different application scenes, arranging theme objects in each application scene, and establishing a logic model of the theme objects and a water conservancy theme object relation table to obtain association relations of the water conservancy theme objects; acquiring water conservancy holographic data under a certain application scene through the association relation between the theme object and the water conservancy theme object; wherein the logic model comprises: any one or more of water conservancy theme object foundation, real-time water rain condition, weather and space dimension information are combined; the water conservancy theme object comprises: any one or more of a river basin, river, lake, reservoir dam, hydrological station, sluice, pump station.
Specifically, the ODS original layer and the DWD detail layer are used for carding and organizing data in a bottom-up mode, the DWS theme layer is used for sorting theme objects in various application scenes according to the traffic flow and the application scene of monitoring, forecasting and early warning of the torrent disasters, and inversely pushing data requirements, a logic model and an object relation table of the theme objects are built, the logic model comprises basic, real-time water rain condition, weather and space dimensional information of the objects, and an application party can conveniently and rapidly acquire holographic data of a certain application scene through object id and association relation.
The topic division refers to the Water conservancy object basic database Table Structure and identifier (SLT 809-2021).
The water conservancy objects related to the project comprise 8 water conservancy domains, rivers, lakes, reservoirs, reservoir dams, hydrological measuring stations, sluice gates and pumping stations. The design of the theme layer takes a water conservancy object as a core, dimensional information such as a foundation, space, real-time water rain conditions and the like of the water conservancy object is combed, and a logic model of the object is built. In addition, an object relation table should be established, and an association relation of objects is established.
S134, the ADS thematic layer is used for carrying out water conservancy data application analysis on different professions according to user requirements, and then organizing a database to design various application themes so as to generate hydrological models in different application scenes.
The ADS thematic layer meets the requirements of specific departments or users, and is oriented to analysis of certain professional applications, the data of the DWD detail layer and the DWS thematic layer are reorganized and built into libraries, the requirements can be set up by each application standard, and the data resource pool responds again.
In this embodiment, the ADS thematic layer is designed for input data obtained by calculation of a hydrological model; the hydrological model comprises: a forecasting model, a one-dimensional evolution model, a two-dimensional evolution model and an early warning model.
Specifically, in all information systems related to the project, hydrologic model calculation is an indispensable link in a mountain torrent and drought disaster 'four-in-one' business scene, so that an ADS thematic layer is designed according to input data requirements of hydrologic model calculation. The hydrological model designed in this time includes: the model is four models of a forecast model, a one-dimensional evolution model, a two-dimensional evolution model and an early warning model, and more models can be expanded subsequently.
The ADS thematic layer can further develop and design various application themes, and each standard section is required to put forward requirements, and the data resource pool responds again. Wherein, the water conservancy object basic database table structure and identifier (SLT 809-2021).
And S14, developing a big data platform based on the data resource pool so as to realize the fine treatment of water conservancy data.
In this embodiment, the development is completed using a big data platform based on the design scheme. Determining a water conservancy data resource pool according to the steps; building a development environment and a water conservancy big data management platform; designing a water conservancy data model; then, a data processing program is correspondingly developed, and a data visualization process is further realized; realizing data sharing; further, the water conservancy data is treated; continuously optimizing and updating databases of big data management platforms, and the like.
Specifically, first, a required data resource pool is determined, including: data type, data format, data source, etc. Building a development environment suitable for large data platform development, comprising: hardware devices, software tools, network environments, and the like. And then, according to the characteristics and service requirements of the water conservancy data, designing a proper data model, wherein the method comprises the following steps: data structure, data relationship, data flow, etc.
Then, developing a data processing program according to the designed data model, including: data acquisition, cleaning, conversion, analysis, and the like.
And then, the processed data is presented through a visualization technology so as to more intuitively know the condition of the data and discover the rules and trends in the data. And sharing the processed data to other business departments or external units to support decision and planning of water conservancy works.
Then, the data is finely managed and controlled by a data management technology, which comprises the following steps: data quality, data security, data lifecycle, etc. And continuously optimizing and updating the big data platform according to the actual application condition and the feedback opinion so as to improve the data processing efficiency and the data quality.
Through the steps, large data platform development can be performed based on the data resource pool, fine treatment of water conservancy data is achieved, and efficiency and accuracy of water conservancy work are improved.
According to the design method for the data management resource pool of the water conservancy industry based on the big data platform, the digital twin water conservancy data resource pool is established, the integrated data service and linkage of cross-business and cross-organization are realized, standard authoritative data is provided for service matters and application matters continuously changed in the digital twin engineering, and a solid foundation is finally provided for building an authoritative data center. Meanwhile, the data management capability in the water conservancy data management process can be improved; the data value is further mined, the data utilization rate is improved, the stability of the data management model is high, the problem of inconsistent sources of water conservancy data in a complex information system environment can be solved, one source is realized, and the fine management of the water conservancy data is realized.
The protection scope of the water conservancy industry data management resource pool design method based on the big data platform is not limited to the step execution sequence listed in the embodiment, and all the schemes implemented by step increase and decrease and step replacement of the prior art according to the principles of the present application are included in the protection scope of the present application.
The present embodiment additionally provides a computer readable storage medium having a computer program stored thereon, which when executed by a processor implements a water conservancy industry data governance resource pool design method based on a big data platform as described in fig. 1.
The present application may be a system, method, and/or computer program product at any possible level of technical detail. The computer program product may include a computer readable storage medium having computer readable program instructions embodied thereon for causing a processor to implement aspects of the present application.
The computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: portable computer disks, hard disks, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static Random Access Memory (SRAM), portable compact disk read-only memory (CD-ROM), digital Versatile Disks (DVD), memory sticks, floppy disks, mechanical coding devices, punch cards or in-groove structures such as punch cards or grooves having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media, as used herein, are not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., optical pulses through fiber optic cables), or electrical signals transmitted through wires.
The computer readable program described herein may be downloaded from a computer readable storage medium to a respective computing/processing device or to an external computer or external storage device via a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers and/or edge servers. The network interface card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium in the respective computing/processing device. Computer program instructions for carrying out operations of the present application may be assembly instructions, instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, c++ or the like and a procedural programming language such as the "C" language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on the computer or entirely on the computer or server. In the case of a computer, the computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (e.g., connected through the internet using an internet service provider). In some embodiments, aspects of the present application are implemented by personalizing electronic circuitry, such as programmable logic circuitry, field Programmable Gate Arrays (FPGAs), or Programmable Logic Arrays (PLAs), with state information for computer readable program instructions, which may execute the computer readable program instructions.
The embodiment of the application also provides a water conservancy industry data treatment resource pool design system based on a big data platform, the water conservancy industry data treatment resource pool design system based on the big data platform can realize the water conservancy industry data treatment resource pool design method based on the big data platform, but the implementation device of the water conservancy industry data treatment resource pool design method based on the big data platform, which is disclosed by the application, comprises but is not limited to the structure of the water conservancy industry data treatment resource pool design system based on the big data platform, and all the structural deformation and replacement of the prior art according to the principles of the application are included in the protection scope of the application.
The water conservancy industry data management resource pool design system based on the big data platform provided by the embodiment will be described in detail below with reference to the drawings.
The embodiment provides a water conservancy industry data management resource pool design system based on big data platform, includes:
referring to fig. 5, a schematic structural diagram of a hydraulic industry data management resource pool design system based on a big data platform according to an embodiment of the present invention is shown. As shown in fig. 5, the hydraulic industry data management resource pool design system based on the big data platform includes: an acquisition module 51, a standard formulation module 52, a design module 53, and a platform development module 54.
The obtaining module 51 is configured to obtain an association relationship and a data flow direction between the systems based on the water conservancy service system.
Based on the water conservancy service system, the association relation and the data flow direction between the systems are acquired, and based on the water conservancy service system, the association relation and the data flow direction between the systems are acquired.
The robot formation in this application is fixed, but its position is uncertain. That is, the initial and arrival positions of the formation are not fixed in a Cartesian coordinate system. Thus, a determination of the coordinates is required.
And (3) carrying out information system investigation by the established investigation group, and acquiring the water conservancy system data information from the data layer.
In this embodiment, an investigation group is established, and according to actual conditions, information system investigation is performed through a plurality of investigation shapes, and water conservancy system data information is obtained from a data layer. Among the various investigation forms include: different forms of interviews, questionnaires, interviews, etc.; the water conservancy system data information comprises: the hydraulic system comprises a hydraulic system function, a hydraulic data production and utilization status, a hydraulic data type, a hydraulic data flow direction and a plurality of hydraulic data information.
And establishing a business data matrix based on the water conservancy system data information.
In this embodiment, a service data matrix is established to provide a basis for designing a data resource pool subsequently.
Specifically, according to the service requirements and objectives, it is determined from which data sources data needs to be acquired, for example: databases, reports, APIs, etc. Formulating a data collection plan based on the determined data sources, comprising: collection range, collection frequency, collection time, etc. of the data. Data is then acquired from the data sources according to the formulated data collection plan and cleaned and consolidated as necessary. According to the business requirements and the targets, a model and a structure of the data matrix are determined, for example, what data are respectively represented by rows and columns of the matrix, association relations among the data and the like. Then, according to the determined data matrix model, filling the collected data into the matrix, and carrying out necessary calculation and processing; after the data matrix is constructed, verification is needed to ensure the accuracy and the integrity of the data; once the data matrix is validated, the data matrix may be published for use by subsequent data analysis and decision support.
And adopting a data exploration tool of a large data platform to automatically scan and inventory.
In this embodiment, for structured data in the service system, a data exploration tool of a large data platform is used to automatically scan and inventory.
Specifically, depending on the traffic requirements and the size of the data volume, a suitable data exploration tool is selected, for example: tableau, powerBI, redash, etc. The selected data exploration tool is then connected to the large data platform to enable storage and analysis of the data. A view of the data is created on a large data platform to more conveniently view and analyze the data. And then using an automatic scanning function in the data exploration tool to automatically scan the structured data so as to find out the abnormality and the problem in the data. And using a data checking function in the data exploration tool to check the structured data so as to know the distribution, trend and association relation of the data. Then, according to the scanning and checking results, the problems and opportunities in the data are analyzed, and corresponding suggestions and measures are provided. Finally, the analysis result is exported to a designated storage device or shared folder for subsequent use.
Through the steps, the structured data in the business system can be automatically scanned and checked by using the data exploration tool of the big data platform, so that enterprises can be helped to better know and utilize the data resources.
The standard making module 52 is configured to make a water conservancy data standard specification based on a national standard and an industry standard.
In the embodiment, a water conservancy data standard specification is determined, and national standards and industry specifications of water conservancy industry are collected; then uniformly making water conservancy data standard; respectively formulating a data sharing specification, a data management specification and an application development specification; and writing a standard specification document, and then issuing a popularization standard specification. In addition, attention is paid to updating of the standard specification.
Specifically, the objectives of the water conservancy data standard specification are specified, for example: standardizing data formats, improving data quality, facilitating data sharing, etc., and determining the applicable scope of standard specifications, for example: applicable to areas, applicable to business fields, etc. Then, the national standard and industry standard related to the water conservancy data, and related technical literature and data are collected for analysis and research. And then, according to the collected data and analysis results, formulating water conservancy data standards including data formats, data types, data precision, data element definitions and the like. Then, formulating a water conservancy data sharing specification, including: sharing mode, sharing flow, sharing protocol, sharing security, etc.; formulating a water conservancy data management specification, comprising: data acquisition, data processing, data storage, data backup and the like; formulating a water conservancy application development specification, comprising: application development flow, application interface design, etc. Next, the established water conservancy data standard specification is written into a document, which comprises the following steps: standard numbers, names, version numbers, release dates, etc., and typesetting and printing according to the document format. Finally, release and popularization standard specifications: publishing and popularizing the written water conservancy data standard specification document, which comprises the following steps: release to related departments of water conservancy industry, popularization to related enterprises and institutions, and the like.
Note that the standard specification is maintained and updated continuously: and continuously maintaining and updating the formulated water conservancy data standard specification according to the actual application condition and the feedback opinion so as to ensure the adaptability and the effectiveness of the water conservancy data standard specification.
The design module 53 is configured to design a data resource pool based on the hydraulic data standard specification.
The data resource pool adopts a typical four-layer design of a data warehouse and is divided into: ODS pastes source layer, DWD detail layer, DWS theme layer, ADS thematic layer.
The ODS is attached to the source layer and used for dividing data fields of water conservancy data.
In this embodiment, the ODS paste source layer includes: and checking, data carding and data sorting of the water conservancy data system.
Specifically, the layer basically does not change the original data, so as to solve the problem of possible data tracing. And dividing the data domain and referring to the data source.
The layer is accessed to all original data of the existing system and the system to be built, and the accessed data is basically not changed and processed, so that the problem of tracing the data source is prevented. The data of the layer corresponds to the data source system. The working focus of the source layer is on system checking and data carding, and data useful for each scale section of the project are tidied.
And the DWD detail layer is used for carrying out data processing on the divided water conservancy data to obtain standardized water conservancy data.
The data is from ODS paste source layer, and is cleaned, integrated and normalized according to standard, rule and the like. Dirty data, garbage data, data with inconsistent specifications, inconsistent state definitions and inconsistent naming standards are processed to provide standard data for the DWS topic layer.
In this embodiment, the data of the DWD detail layer is derived from the ODS paste source layer; the DWD detail layer includes: carrying out processes of data cleaning, data domain division, data fusion and specification on the water conservancy data according to the standards and rules; the data domain partitioning includes: any one or more combination of basic data, space remote sensing information, real-time water and rain condition monitoring, weather forecast and business data; the data cleaning is used for providing specific and definite data requirements and reference standard specifications by a user, determining cleaning rules, feeding back to the data center station and the data source, and improving the data quality.
Specifically, the DWD layer performs cleaning, integration, and standardization according to standards, rules, and the like. The design emphasis is on data domain partitioning and the formulation of data cleansing criteria.
The data domain is divided, and the technical requirement (trial) of a provincial mountain torrent disaster monitoring, forecasting and early warning platform is referred to (9 months in 2020). The method is divided into basic data types, space remote sensing information types, real-time water and rain condition monitoring types, weather forecast types and business data types.
In addition, the data cleaning standard refers to a real-time rain condition database table structure and an identifier standard (SL 323-2011), a mountain torrent disaster investigation and evaluation database table structure and an identifier standard (2015, 7 months), a flood control project database table structure and an identifier (2015, 2 months), and a real-time rain condition database table structure and an identifier standard (SL 323-2011).
The ODS original layer and the DWD detail layer are used for carding and organizing data in a bottom-up mode, and the DWS theme layer is used for monitoring mountain torrent disasters.
And the DWS topic layer is used for fusing the standardized water conservancy data to obtain a service data table of a topic domain.
Based on the DWD detail layer data, service data of a certain topic domain is integrated and summarized and analyzed, typically in a broad table.
In the embodiment, a water conservancy service data table for summarizing and analyzing a certain topic domain in the water profit industry is integrated based on DWD detail layer data and user requirements; according to different application scenes, arranging theme objects in each application scene, and establishing a logic model of the theme objects and a water conservancy theme object relation table to obtain association relations of the water conservancy theme objects; acquiring water conservancy holographic data under a certain application scene through the association relation between the theme object and the water conservancy theme object; wherein the logic model comprises: any one or more of water conservancy theme object foundation, real-time water rain condition, weather and space dimension information are combined; the water conservancy theme object comprises: any one or more of a river basin, river, lake, reservoir dam, hydrological station, sluice, pump station.
The topic division refers to the Water conservancy object basic database Table Structure and identifier (SLT 809-2021).
The water conservancy objects related to the project comprise 8 water conservancy domains, rivers, lakes, reservoirs, reservoir dams, hydrological measuring stations, sluice gates and pumping stations. The design of the theme layer takes a water conservancy object as a core, dimensional information such as a foundation, space, real-time water rain conditions and the like of the water conservancy object is combed, and a logic model of the object is built. In addition, an object relation table should be established, and an association relation of objects is established.
The ADS thematic layer is used for carrying out water conservancy data application analysis and reorganization and library building for different professions according to user requirements so as to design various application themes and generate hydrological models in different application scenes.
The ADS thematic layer meets the requirements of specific departments or users, and is oriented to analysis of certain professional applications, the data of the DWD detail layer and the DWS thematic layer are reorganized and built into libraries, the requirements can be set up by each application standard, and the data resource pool responds again.
In this embodiment, the ADS thematic layer is designed for input data obtained by calculation of a hydrological model; the hydrological model comprises: a forecasting model, a one-dimensional evolution model, a two-dimensional evolution model and an early warning model.
Specifically, in all information systems related to the project, hydrologic model calculation is an indispensable link in a mountain torrent and drought disaster 'four-in-one' business scene, so that an ADS thematic layer is designed according to input data requirements of hydrologic model calculation. The hydrological model designed in this time includes: the model is four models of a forecast model, a one-dimensional evolution model, a two-dimensional evolution model and an early warning model, and more models can be expanded subsequently.
The ADS thematic layer can further develop and design various application themes, and each standard section is required to put forward requirements, and the data resource pool responds again. Water conservancy object basic database Table Structure and identifier (SLT 809-2021).
The platform development module 54 is used for carrying out large data platform development based on the data resource pool so as to realize the fine management of water conservancy data.
And developing a big data platform based on the data resource pool so as to realize the fine treatment of water conservancy data.
In this embodiment, the development is completed using a big data platform based on the design scheme. Determining a water conservancy data resource pool according to the steps; building a development environment and a water conservancy big data management platform; designing a water conservancy data model; then, a data processing program is correspondingly developed, and a data visualization process is further realized; realizing data sharing; further, the water conservancy data is treated; continuously optimizing and updating databases of big data management platforms, and the like.
Specifically, first, a required data resource pool is determined, including: data type, data format, data source, etc. Building a development environment suitable for large data platform development, comprising: hardware devices, software tools, network environments, and the like. And then, according to the characteristics and service requirements of the water conservancy data, designing a proper data model, wherein the method comprises the following steps: data structure, data relationship, data flow, etc.
Then, developing a data processing program according to the designed data model, including: data acquisition, cleaning, conversion, analysis, and the like.
And then, the processed data is presented through a visualization technology so as to more intuitively know the condition of the data and discover the rules and trends in the data. And sharing the processed data to other business departments or external units to support decision and planning of water conservancy works.
Then, the data is finely managed and controlled by a data management technology, which comprises the following steps: data quality, data security, data lifecycle, etc. And continuously optimizing and updating the big data platform according to the actual application condition and the feedback opinion so as to improve the data processing efficiency and the data quality.
Through the steps, large data platform development can be performed based on the data resource pool, fine treatment of water conservancy data is achieved, and efficiency and accuracy of water conservancy work are improved.
A water conservancy industry data management resource pool design system based on a big data platform realizes cross-service and cross-organization integrated data service and linkage by establishing a digital twin water conservancy data resource pool, provides standard authoritative data for service matters and application matters continuously changed in digital twin engineering, and finally provides a solid foundation for building an authoritative data center.
It should be noted that, it should be understood that the division of the modules of the above system is merely a division of a logic function, and may be fully or partially integrated into a physical entity or may be physically separated. And these modules may all be implemented in software in the form of calls by the processing element; or can be realized in hardware; the method can also be realized in a form of calling software by a processing element, and the method can be realized in a form of hardware by a part of modules. For example, the x module may be a processing element that is set up separately, may be implemented in a chip of the system, or may be stored in a memory of the system in the form of program code, and the function of the x module may be called and executed by a processing element of the system. The implementation of the other modules is similar. In addition, all or part of the modules can be integrated together or can be independently implemented. The processing element described herein may be an integrated circuit having signal processing capabilities. In implementation, each step of the above method or each module above may be implemented by an integrated logic circuit of hardware in a processor element or an instruction in a software form.
The above modules may be one or more integrated circuits configured to implement the above methods, for example: one or more application specific integrated circuits (Application Specific Integrated Circuit, abbreviated as ASIC), or one or more microprocessors (digital signal processor, abbreviated as DSP), or one or more field programmable gate arrays (FieldProgrammable Gate Array, abbreviated as FPGA), or the like. For another example, when a module above is implemented in the form of a processing element scheduler code, the processing element may be a general-purpose processor, such as a central processing unit (Central Processing Unit, CPU) or other processor that may invoke the program code. For another example, the modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).
Referring to fig. 6, a schematic structural diagram of a hydraulic industry data management resource pool design device based on a big data platform according to an embodiment of the present invention is shown. As shown in fig. 6, the present embodiment provides a water conservancy industry data management resource pool design device based on a big data platform, where the water conservancy industry data management resource pool design device based on the big data platform includes: a processor 61 and a memory 62; the memory 62 is used for storing a computer program; the processor 61 is connected to the memory 62, and is configured to execute a computer program stored in the memory 62, so that the hydraulic industry data management resource pool design device based on the big data platform performs the steps of the hydraulic industry data management resource pool design method based on the big data platform as described above.
Preferably, the memory may comprise random access memory (RandomAccess Memory, abbreviated as RAM), and may further comprise non-volatile memory (non-volatile memory), such as at least one magnetic disk memory.
The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (Network Processor, NP for short), etc.; but also digital signal processors (Digital Signal Processing, DSP for short), application specific integrated circuits (Application Specific Integrated Circuit, ASIC for short), field programmable gate arrays (Field Programmable GateArray, FPGA for short) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
In summary, the water conservancy industry data management resource pool design method based on the big data platform has the following beneficial effects:
according to the design method for the data management resource pool of the water conservancy industry based on the big data platform, the digital twin water conservancy data resource pool is established, the integrated data service and linkage of cross-business and cross-organization are realized, standard authoritative data is provided for service matters and application matters continuously changed in the digital twin engineering, and a solid foundation is finally provided for building an authoritative data center. Meanwhile, the data management capability in the water conservancy data management process can be improved; the data value is further mined, the data utilization rate is improved, the stability of the data management model is high, the problem of inconsistent sources of water conservancy data in a complex information system environment can be solved, one source is realized, and the fine management of the water conservancy data is realized.
The above embodiments are merely illustrative of the principles of the present invention and its effectiveness, and are not intended to limit the invention. Modifications and variations may be made to the above-described embodiments by those skilled in the art without departing from the spirit and scope of the invention. Accordingly, it is intended that all equivalent modifications and variations of the invention be covered by the claims, which are within the ordinary skill of the art, be within the spirit and scope of the present disclosure.
Claims (10)
1. A water conservancy industry data management resource pool design method based on a big data platform is characterized by comprising the following steps:
based on a water conservancy service system, acquiring an association relation and a data flow direction between the systems;
establishing a water conservancy data standard specification based on national standards and industry standards;
designing a data resource pool based on the water conservancy data standard specification;
and developing a big data platform based on the data resource pool so as to realize the fine treatment of water conservancy data.
2. The method for designing the water conservancy industry data management resource pool based on the big data platform as set forth in claim 1, wherein the obtaining of the association relation and the data flow direction between the systems based on the water conservancy business system comprises the following steps:
Establishing an investigation group to carry out information system investigation and acquire water conservancy system data information from a data layer;
establishing a business data matrix based on the water conservancy system data information;
and adopting a data exploration tool of a large data platform to automatically scan and inventory.
3. The method for designing a water conservancy industry data management resource pool based on a big data platform according to claim 2, wherein the investigation form comprises: any one or more combinations of walks, questionnaires, interviews;
the water conservancy system data information comprises: any one or more of water conservancy system functions, water conservancy data production and utilization status, water conservancy data types and water conservancy data flow directions.
4. The method for designing a water conservancy industry data management resource pool based on a big data platform as claimed in claim 1, wherein the data resource pool adopts a data warehouse four-layer design structure, and the structure comprises: ODS pastes source layer, DWD detail layer, DWS theme layer and ADS thematic layer;
the ODS is attached to the source layer and is used for dividing data fields of water conservancy data;
the DWD detail layer is used for carrying out data processing on the divided water conservancy data to obtain standardized water conservancy data;
The DWS topic layer is used for fusing the standardized water conservancy data to obtain a service data table of a topic domain;
the ADS thematic layer is used for carrying out water conservancy data application analysis and reorganization and library building for different professions according to user requirements so as to design various application themes and generate hydrological models in different application scenes.
5. The method for designing a water conservancy industry data management resource pool based on a big data platform as set forth in claim 4, wherein the ODS paste source layer includes: and checking, data carding and data sorting of the water conservancy data system.
6. The method for designing a water conservancy industry data management resource pool based on a big data platform as set forth in claim 4, wherein the data of the DWD detail layer is derived from the ODS paste source layer;
the DWD detail layer includes: carrying out processes of data cleaning, data domain division, data fusion and specification on the water conservancy data according to the standards and rules;
the data domain partitioning includes: any one or more combination of basic data, space remote sensing information, real-time water and rain condition monitoring, weather forecast and business data;
the data cleaning is used for providing specific and definite data requirements and reference standard specifications by a user, determining cleaning rules, feeding back to the data center station and the data source, and improving the data quality.
7. The method for designing a water conservancy industry data management resource pool based on a big data platform as set forth in claim 4, wherein the DWS theme layer includes the steps of:
based on DWD detail layer data and user requirements, integrating and analyzing a water conservancy service data table of a certain topic domain in the water profit industry;
according to different application scenes, arranging theme objects in each application scene, and establishing a logic model of the theme objects and a water conservancy theme object relation table to obtain association relations of the water conservancy theme objects;
acquiring water conservancy holographic data under a certain application scene through the association relation between the theme object and the water conservancy theme object;
wherein the logic model comprises: any one or more of water conservancy theme object foundation, real-time water rain condition, weather and space dimension information are combined;
the water conservancy theme object comprises: any one or more of a river basin, river, lake, reservoir dam, hydrological station, sluice, pump station.
8. The method for designing a water conservancy industry data management resource pool based on a big data platform as set forth in claim 4, wherein the ADS thematic layer is designed for input data obtained by calculation of a hydrological model;
The hydrological model comprises: a forecasting model, a one-dimensional evolution model, a two-dimensional evolution model and an early warning model.
9. Water conservancy industry data governance resource pool design system based on big data platform, characterized by comprising:
the acquisition module is used for acquiring the association relation and the data flow direction among the systems based on the water conservancy service system;
the standard making module is used for making water conservancy data standard specifications based on national standards and industry standards;
the design module is used for designing a data resource pool based on the water conservancy data standard specification;
and the platform development module is used for carrying out large data platform development based on the data resource pool so as to realize the fine treatment of water conservancy data.
10. Water conservancy industry data governance resource pool design device based on big data platform, its characterized in that includes: a processor and a memory;
the memory is used for storing a computer program;
the processor is connected with the memory, and is used for executing the computer program stored in the memory, so that the water conservancy industry data management resource pool design device based on the big data platform executes the water conservancy industry data management resource pool design method based on the big data platform according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311547828.4A CN117473024A (en) | 2023-11-20 | 2023-11-20 | Water conservancy industry data management resource pool design method based on big data platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311547828.4A CN117473024A (en) | 2023-11-20 | 2023-11-20 | Water conservancy industry data management resource pool design method based on big data platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117473024A true CN117473024A (en) | 2024-01-30 |
Family
ID=89625478
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311547828.4A Pending CN117473024A (en) | 2023-11-20 | 2023-11-20 | Water conservancy industry data management resource pool design method based on big data platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117473024A (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113626648A (en) * | 2021-07-30 | 2021-11-09 | 联通(广东)产业互联网有限公司 | Water conservancy data processing system, method and storage medium |
| CN114647683A (en) * | 2022-05-07 | 2022-06-21 | 烟台磐能电气控制系统有限公司 | Intelligent water affair comprehensive management system |
-
2023
- 2023-11-20 CN CN202311547828.4A patent/CN117473024A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113626648A (en) * | 2021-07-30 | 2021-11-09 | 联通(广东)产业互联网有限公司 | Water conservancy data processing system, method and storage medium |
| CN114647683A (en) * | 2022-05-07 | 2022-06-21 | 烟台磐能电气控制系统有限公司 | Intelligent water affair comprehensive management system |
Non-Patent Citations (2)
| Title |
|---|
| 姜仕琪: "辽宁省信息系统共享整合总体技术架构", 《水利技术监督》, 7 April 2022 (2022-04-07), pages 70 - 74 * |
| 马花月等: "《大数据技术在智慧工程中的研究和应用》", 《水利规划与设计》, vol. 2021, no. 10, 18 September 2021 (2021-09-18), pages 49 - 53 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110795509B (en) | Method and device for constructing index blood-margin relation graph of data warehouse and electronic equipment | |
| Xu et al. | Optimization algorithms for construction site layout planning: a systematic literature review | |
| CN102024204B (en) | Constructing method of reliability design analysis service system of service oriented architecture | |
| CN102426582B (en) | Data manipulation management devices and data manipulation management method | |
| CN106407278A (en) | Architecture design system of big data platform | |
| CN115186629B (en) | Coding method, coding system, coding terminal and coding medium of hydraulic and hydroelectric engineering management system | |
| CN103336791A (en) | Hadoop-based fast rough set attribute reduction method | |
| US11875412B2 (en) | System for fast composing, launch and configuration of customizable second-tier transfer structures with build-in auditing and monitoring structures and method thereof | |
| CN106503811A (en) | A kind of infrastructure full life cycle management method based on big data | |
| CN114328980A (en) | Knowledge graph construction method and device combining RPA and AI, terminal and storage medium | |
| CN115309749A (en) | Big data experiment system for scientific and technological service | |
| CN115577646A (en) | Data modeling method, device, equipment and medium based on multi-source heterogeneous data | |
| CN113722564A (en) | Visualization method and device for energy and material supply chain based on space map convolution | |
| CN117473024A (en) | Water conservancy industry data management resource pool design method based on big data platform | |
| CN106157216A (en) | A kind of spatial information data management method and system | |
| Mansoor et al. | Scientometric analysis and critical review on the application of deep learning in the construction industry | |
| CN115794038A (en) | Software engineering development system based on Internet and method thereof | |
| Alwaisi et al. | A review on big data stream processing applications: contributions, benefits, and limitations | |
| Xue et al. | Information enterprise architecture for smart transportation system | |
| Takkala et al. | CHIP: Clustering hotspots in layout using integer programming | |
| CN112527870B (en) | Electronic report generation method, device, electronic equipment and storage medium | |
| Planting | Developing a data repository for the Climate Adaptive City Enschede | |
| CN115422193B (en) | Storage method, system and medium for multi-source data of city information model | |
| CN109670601B (en) | Machine learning feature generation method and device, electronic device and storage medium | |
| Liu et al. | Service oriented cross-platform framework for 3D part library based on diversiform knowledge implantation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |