CN112579563B - Power grid big data-based warehouse visualization modeling system and method - Google Patents
Power grid big data-based warehouse visualization modeling system and method Download PDFInfo
- Publication number
- CN112579563B CN112579563B CN202011295978.7A CN202011295978A CN112579563B CN 112579563 B CN112579563 B CN 112579563B CN 202011295978 A CN202011295978 A CN 202011295978A CN 112579563 B CN112579563 B CN 112579563B
- Authority
- CN
- China
- Prior art keywords
- data
- model
- management module
- database
- logic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/211—Schema design and management
- G06F16/212—Schema design and management with details for data modelling support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention relates to the technical field of big data, in particular to a power grid big data-based visual modeling system for a plurality of bins, which comprises a data source configuration module, a hierarchy/theme domain management module, a logic model design management module, a physical model management module and a table management module, wherein the data source configuration module is used for configuring a specified data source, the hierarchy/theme domain management module is used for establishing the hierarchical relation of a database, the logic model design management module is used for forming a logic model, the physical model management module is used for forming a physical model, and the table management module is used for forming the physical model into an actual table to maintain the database table. The warehouse-counting visual modeling system and method based on the big data of the power grid can quickly and efficiently construct a database modeling platform meeting the requirements, and improve the database modeling efficiency and the data modeling normalization of the big data center of the power grid.
Description
Technical Field
The invention relates to the technical field of big data, in particular to a warehouse counting visual modeling system and method based on power grid big data.
Background
The data warehouse is a warehouse which organizes, stores and manages data by adopting computer storage equipment according to the characteristics of data types, structures and the like, can be regarded as a file cabinet of the data in nature, and can perform management operations such as adding, intercepting, updating, deleting and the like on the data in the warehouse according to the requirements of users. With the rapid development of social economy and the arrival of a big data era, the requirements of the power industry on the storage, management and analysis of the big power data are higher and higher, and different data warehouses are required to be established according to different types of big power data, so that the big power data warehouse is inevitably constructed for power enterprises.
At present, the service volume of a power grid is very complex, the data volume is huge, and data application gradually becomes the main development direction of the power grid, so that the dependence of mass data precipitated by multiple power grids is stronger and stronger, and a correct and coherent data flow plays a decisive role in making a quick and flexible decision for a power industry decision-making department. The best result can be guaranteed only by establishing correct data flow and data structures, so that a set of smooth and efficient warehouse modeling system for the large data of the power grid is particularly important to establish.
Patent specification No. 201811384906.2 discloses a visualization-based big data warehouse design method and system, the system includes a visualization library construction table information maintenance unit, a JAVA service unit and a Hive data warehouse; the method comprises the steps of firstly adopting a visual operation interface to build database table building information and adopting a JAVA service unit to determine a specific operation type, then packaging the database table building information into a Hive data warehouse which can be obviously executed, connecting the JAVA service unit to a Hive server2 for service, and finally executing commands in the Hive data warehouse, and creating a database and a data table. The method provides a visual database building and table building data maintenance interface, simplifies the operation modes of new construction, editing, deleting and modifying of the large data warehouse, reduces the technical difficulty and the learning cost, and realizes the monitoring and management of the data warehouse. The method mainly aims at the management of a large data warehouse, and does not relate to the improvement of the visual modeling efficiency of the warehouse and the normalization of data modeling.
Disclosure of Invention
The invention provides a power grid big data-based data-obtaining bin visual modeling system which can improve the modeling efficiency of a power grid big data center database and the normalization of data modeling.
The invention further aims to provide a digital bin visualization modeling method based on the big data of the power grid.
In order to achieve the above purpose, the technical means adopted is as follows:
a warehouse-counting visual modeling system based on power grid big data comprises a data source configuration module, a hierarchy/theme domain management module, a logic model design management module, a physical model management module and a table management module; the data configuration module and the hierarchy/theme domain management module are both connected with a logical model design management module, the logical model design management module is connected with a physical model management module, and the physical model management module is connected with a table management module; the data source configuration module is used for configuring a specified data source; the hierarchy/subject domain management module establishes a hierarchy relation of a database; inputting the hierarchical relationship between the designated data source and the database into the logic model design management module; the logic model design management module forms a logic model by utilizing the hierarchical relation between the specified data source and the database; the logic model is used as the input of the physical model management module; the physical model management module forms a physical model by using the logic model; the physical model is used as the input of the table management module; and the table management module forms the physical model into an actual table to realize maintenance of the database table.
The configuration of a designated data source can be realized through the data source configuration module, the establishment of the hierarchical relation of the database can be realized through the hierarchy/theme domain management module, the formation of a logic model can be realized through the logic model design management module, the formation of a physical model can be realized through the physical model management module, and the physical model can be formed into an actual table through the table management module so as to realize the maintenance of the database table.
Further, the data source configuration module realizes the configuration of the specified data source; the hierarchy/theme and management module realizes the establishment of a data warehouse hierarchy relationship, wherein each hierarchy comprises a source layer, an integration layer, a theme layer and a special topic layer; the logic model design management module can realize the reverse construction of a logic model of a user-defined visual model design and a database model by utilizing the specified data source and the warehouse hierarchical relation; the design of the self-defined visual model supports the creation of database logic model test questions and field attributes thereof on a design panel and supports the export of a model from a logic model design interface to be provided for developers to develop data application or import a model created by other tools, wherein the other tools comprise a PowerDesigner enterprise modeling tool and the like; the reverse construction logic model of the database model automatically extracts metadata information of the database table from a source database through the logic model design management module, automatically generates the logic model of the database table, supports manual adjustment and does not need to manually create the database model, wherein the metadata refers to the attribute of the database table; the physical model management module realizes the construction of a physical model, and generates a database execution script DDL statement by using a logical model generated by the logical model design management module as the input of the physical model management module, wherein the metadata of the physical model is changed along with the change of the logical model, the change condition can be tracked according to the version number, and the DDL refers to a database table creation script; the table management module generates an executable database script through the physical model generated by the physical model management module, automatically executes the database script, creates an actual table and realizes the maintenance of the database table.
Furthermore, the logic model design management module also comprises a visual data modeling tool module, a visual database modeling tool module and a reverse engineering module; the visual data modeling tool module is connected with the visual database modeling tool module, and the visual database modeling tool module is connected with the reverse engineering module; and the visual data modeling tool module transmits the change metadata to the visual database modeling module and transmits the large platform data capable of reversely generating the logic model to the reverse engineering module.
The establishment of a logic model design management module is realized through the mutual combination of the visual data modeling tool module, the visual database modeling tool module and the reverse engineering module, so that the logic model function is realized.
A modeling method of a power grid big data-based warehouse visualization modeling system comprises the following steps:
s1, a data warehouse layering rule is formulated by a data warehouse engineer according to business requirements and metadata conditions, and the formulated data warehouse layering rule is transmitted to a logic model design management module;
s2, accessing the big data platform data to a data source configuration module to obtain metadata, and transmitting the metadata to a logic model design management module;
s3, generating a data warehouse layering rule and a metadata generation logic model by using the warehouse design in the step S1 and the metadata in the step S2 through a logic model design management module, and transmitting the logic model to a layer/theme domain management module;
s4, the layer/subject domain management module generates a complete power data warehouse logical model of each layer by using the logical model in the step S3, and transmits the complete power data warehouse logical model of each layer to the physical model management module;
s5, the physical model management module generates a physical model by using the complete power data warehouse logical model of each layer in the step S4, and transmits the physical model to the table management module;
s6, the table management module realizes automatic synchronous recovery of data of the data table by using the physical model in the step S5;
and S7, exporting a data model of the designated data, wherein the data model comprises a data structure, data operation and data constraint.
The database is modeled in a visual mode, a more efficient operation mode is provided for a data warehouse engineer, the complexity of data modeling design is reduced, and functions such as processing the database are facilitated, so that the design efficiency of the data warehouse is improved.
Further, the specific process of step S1 is: and planning the construction scheme of the whole data warehouse by combining the power grid source data structure and the specific output requirement, and formulating a data warehouse design rule by a data warehouse engineer.
The design rule of the data warehouse is set by a warehouse engineer, and different rules can be set for different databases of the system, so that the planning of different databases is realized.
Further, the specific process of step S2 is: the big data platform is accessed to a required database table to provide metadata for creating a data logic model.
Through a large data platform, a large amount of metadata can be obtained, and the metadata is provided for the data logic model.
Further, the specific process of step S3 is:
s3.1: opening a visual data modeling tool, and realizing the design of a data model through a mode without code dragging; automatically detecting the change of the metadata through the mapping relation between the metadata and the source data structure; metadata change of the data physical model is realized in a version control mode;
s3.2: opening a visual database modeling interface, and judging whether a logic model can be reversely generated through a grounded data structure of the large data platform according to design planning;
s3.3: extracting database table metadata from a big data platform;
s3.4: generating a logic model by using a reverse engineering building module, selecting a database table specified by a large data platform through a patent tool reverse engineering component, building the reverse engineering model of the database table, generating a logic entity and attributes, and automatically building a direct logic relation of the database table according to a main foreign key relation of the database table;
s3.5: and creating a data table logic entity, creating the entity and the attribute thereof in a way of self-defining design logic design, and establishing an association relation with other logic test questions in the logic model.
The detection and the change of the metadata are realized through a visual data modeling tool, the establishment of a logic model is realized through a visual database modeling interface and a reverse engineering building module, and then the automatic establishment of the incidence relation between a database table and a table in the logic model is realized.
Further, the specific process of step S4 is: according to the design plan of a warehouse engineer, establishing a data model of each layer of a data warehouse in an automatic generation mode, and adjusting entity attribute information through a graphical interface; and finally, generating a complete power data warehouse logic model of each level.
And a complete power data warehouse logic model of each level is established through the layer/subject domain management module, and a logic model is provided for the physical model.
Further, the specific process of step S5 is:
s5.1: generating a data physical model, selecting a logic model in a physical model interface, generating a pseudo code script, automatically executing the script and generating the physical model;
s5.2: establishing a mapping relation between metadata and a source library table, establishing a mapping relation between the metadata of a model entity and the source library table, binding a version number, and providing a relational support for automatic change of the data entity or attribute of the model;
s5.3: detecting metadata change, namely automatically detecting whether metadata of a source database table has a change condition in the data acquisition process of a big data platform through the relationship established between the metadata of a model entity and the source database table, if the metadata is detected to have a change, automatically creating a standby data table by a system without influencing data acquisition, automatically completing the change of a data model by the system, and recording the change content and the version;
s5.4: and tracing metadata change, and inquiring metadata change records of each version according to the version number.
The physical model is formed through the physical model management module, dynamic checking and updating of system metadata are achieved, and change information is recorded.
Preferably, the specific process of step S6 is: and changing the physical model entity, automatically generating script codes after changing, realizing physical model change, and realizing automatic synchronous recovery of the standby data table data.
The automatic synchronous recovery of the data table can be realized by changing the physical model entity.
Further, each layer data model described in step S4 includes an ODS layer model, a DW layer model and a DM layer model, where ODS refers to the data warehouse posting layer, DW refers to the data warehouse integration layer, and DM refers to the data mart layer.
Compared with the prior art, the technical scheme of the invention has the beneficial effects that:
(1) a visual mode is provided for a database modeling interface, the complexity involved in database modeling is reduced, the functions of creating, modifying and deleting the database model are simplified, and the design efficiency is improved.
(2) The function of reversely generating the logic model by the entity table is provided, and the automatic establishment of the incidence relation between the database table and the table in the logic model, the dynamic inspection and the updating of the metadata, the recording of the change information and the unified management of the logic model, the physical model and the database table are realized.
Drawings
FIG. 1 is a functional relationship diagram of a power grid big data-based digital warehouse visual modeling system of the invention;
FIG. 2 is a flow chart of a visualization modeling method of the present invention.
Detailed Description
The drawings are for illustrative purposes only and are not to be construed as limiting the patent;
for the purpose of better illustrating the embodiments, certain features of the drawings may be omitted, enlarged or reduced, and do not represent the size of an actual product;
it will be understood by those skilled in the art that certain well-known structures in the drawings and descriptions thereof may be omitted.
The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.
Example 1
As shown in fig. 1, a power grid big data-based warehouse visualization modeling system includes a data source configuration module, a hierarchy/theme domain management module, a logical model design management module, a physical model management module, and a table management module; the data configuration module and the hierarchy/theme domain management module are both connected with a logical model design management module, the logical model design management module is connected with a physical model management module, and the physical model management module is connected with a table management module; the data source configuration module is used for configuring a specified data source; the hierarchy/subject domain management module establishes a hierarchy relation of a database; inputting the hierarchical relationship between the designated data source and the database into the logic model design management module; the logic model design management module forms a logic model by utilizing the hierarchical relation between the specified data source and the database; the logic model is used as the input of the physical model management module; the physical model management module forms a physical model by using the logic model; the physical model is used as the input of the table management module; and the table management module forms the physical model into an actual table to realize maintenance of the database table.
Meanwhile, the data source configuration module realizes the configuration of the specified data source; the hierarchy/theme and management module realizes the establishment of a data warehouse hierarchy relationship, wherein each hierarchy comprises a source layer, an integration layer, a theme layer and a special topic layer; the logic model design management module can realize the reverse construction of a logic model of a user-defined visual model design and a database model by utilizing the specified data source and the warehouse hierarchical relation; the design of the self-defined visual model supports the creation of database logic model test questions and field attributes thereof on a design panel and supports the export of a model from a logic model design interface to be provided for developers to develop data application or import a model created by other tools, wherein the other tools comprise a PowerDesigner enterprise modeling tool and the like; the reverse construction logic model of the database model automatically extracts metadata information of the database table from a source database through the logic model design management module, automatically generates the logic model of the database table, supports manual adjustment and does not need to manually create the database model, wherein the metadata refers to the attribute of the database table; the physical model management module realizes the construction of a physical model, and generates a database execution script DDL statement by using a logical model generated by the logical model design management module as the input of the physical model management module, wherein the metadata of the physical model is changed along with the change of the logical model, the change condition can be tracked according to the version number, and the DDL refers to a database table creation script; the table management module generates an executable database script through the physical model generated by the physical model management module, automatically executes the database script, creates an actual table and realizes the maintenance of the database table.
Meanwhile, the logic model design management module also comprises a visual data modeling tool module, a visual database modeling tool module and a reverse engineering module; the visual data modeling tool module is connected with the visual database modeling tool module, and the visual database modeling tool module is connected with the reverse engineering module; and the visual data modeling tool module transmits the change metadata to the visual database modeling module and transmits the large platform data capable of reversely generating the logic model to the reverse engineering module.
Example 2
As shown in fig. 2, a warehouse visualization modeling method based on grid big data specifically comprises the following processes:
step 1: a data warehouse layering rule is formulated by a data warehouse engineer according to the service requirement and the metadata condition, and the formulated data warehouse layering rule is transmitted to a logic model design management module;
step 2: accessing the big data platform data to a data source configuration module to obtain metadata, and transmitting the metadata to a logic model design management module;
and step 3: the logic model design management module generates a data warehouse hierarchical rule and a metadata generation logic model by using the warehouse design in the step S1 and the metadata in the step S2, and transmits the logic model to the hierarchy/theme domain management module;
and 4, step 4: the layer/subject domain management module generates a complete power data warehouse logical model of each layer by using the logical model in the step S3, and transmits the complete power data warehouse logical model of each layer to the physical model management module;
and 5: the physical model management module generates a physical model by using the complete power data warehouse logical model of each layer in the step S4, and transmits the physical model to the table management module;
step 6: the table management module realizes automatic synchronous recovery of data of the data table by using the physical model in the step S5;
and 7: a data model specifying the data is derived, wherein the data model includes data structures, data operations, data constraints.
Meanwhile, the specific process of the step 1 is as follows: the method comprises the steps of planning an overall data warehouse construction scheme by combining a database table structure in a power grid source database and specifically outputting data application requirements, and formulating a data warehouse design rule by a data warehouse engineer, wherein the data application comprises power consumption statistical analysis, regional power consumption statistical analysis and the like.
Meanwhile, the specific process of the step 2 is as follows: the big data platform is accessed to a required database table to provide metadata for creating a data logic model.
Meanwhile, the specific process of the step 3 is as follows:
(1) entering a visual data modeling tool module, and realizing the design of a data model through a mode without code dragging; automatically detecting the change of the metadata through the mapping relation between the metadata and the source data structure; metadata change of the data physical model is realized in a version control mode;
(2) entering a visual database modeling tool module, and determining whether a logic model is reversely generated through a database table on which a large data platform falls to the ground by a data warehouse construction scheme;
(3) extracting database table metadata in a logic model design module;
(4) generating a logic model by using a reverse engineering building module in a logic model design module, selecting a database table specified by a large data platform through a patent tool reverse engineering component, building a reverse engineering model of the database table, generating a logic entity and attributes, and automatically building a direct logic relation of the database table according to a main foreign key relation of the database table;
(5) and creating a data table logic entity, creating the entity and the attribute thereof in a way of self-defining design of logic test questions, and establishing an association relation with other logic test questions in the logic model.
Meanwhile, the specific process of the step 4 is as follows: according to a data warehouse layering rule formulated by a data warehouse engineer, establishing a data model of each layer of the data warehouse in an automatic generation mode, and adjusting entity attribute information through a graphical interface; and finally, generating a complete power data warehouse logic model of each level.
Meanwhile, the specific process of the step 5 is as follows:
(1) generating a data physical model, selecting a logic model in a physical model interface, generating a pseudo code script, automatically executing the script and generating the physical model;
(2) establishing a mapping relation between metadata and a source library table, establishing a mapping relation between the metadata of a model entity and the source library table, binding a version number, and providing a relational support for automatic change of the data entity or attribute of the model;
(3) detecting metadata change, namely automatically detecting whether metadata of a source database table has a change condition in the data acquisition process of a big data platform through the relationship established between the metadata of a model entity and the source database table, if the metadata is detected to have a change, automatically creating a standby data table by a system without influencing data acquisition, automatically completing the change of a data model by the system, and recording the change content and the version;
(4) and tracing metadata change, and inquiring metadata change records of each version according to the version number.
Meanwhile, the specific process of the step 6 is as follows: and changing the physical model entity, automatically generating script codes after changing, realizing physical model change, and realizing automatic synchronous recovery of the standby data table data.
The terms describing positional relationships in the drawings are for illustrative purposes only and are not to be construed as limitations of the present patent; it should be understood that the above-described embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Other variations and modifications will be apparent to persons skilled in the art in light of the above description. And are neither required nor exhaustive of all embodiments. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present patent claims.
Meanwhile, each layer of data model described in step 4 includes an ODS layer model, a DW layer model and a DM layer model, where ODS refers to a data warehouse source layer, DW refers to a data warehouse integration layer, and DM refers to a data mart layer.
The positional relationships depicted in the drawings are for illustrative purposes only and are not to be construed as limiting the present patent;
it should be understood that the above-described embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Other variations and modifications will be apparent to persons skilled in the art in light of the above description. And are neither required nor exhaustive of all embodiments. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the claims of the present invention.
Claims (9)
1. A warehouse-counting visual modeling system based on power grid big data is characterized by comprising a data source configuration module, a hierarchy/theme domain management module, a logic model design management module, a physical model management module and a table management module; the data source configuration module and the hierarchy/theme domain management module are both connected with a logic model design management module, the logic model design management module is connected with a physical model management module, and the physical model management module is connected with a table management module; the data source configuration module is used for configuring a specified data source; the hierarchy/subject domain management module establishes a hierarchy relation of a database; inputting the hierarchical relationship between the designated data source and the database into the logic model design management module; the logic model design management module forms a logic model by utilizing the hierarchical relation between the specified data source and the database; the logic model is used as the input of the physical model management module; the physical model management module forms a physical model by using the logic model; the physical model is used as the input of the table management module; the table management module forms the physical model into an actual table to realize maintenance of the database table;
the data source configuration module realizes the configuration of a specified data source; the hierarchy/theme domain management module realizes the establishment of a data warehouse hierarchy relationship, wherein each hierarchy comprises a source layer, an integration layer, a theme layer and a special topic layer; the logic model design management module can realize the reverse construction of a logic model of a user-defined visual model design and a database model by utilizing the specified data source and the warehouse hierarchical relation; the design of the self-defined visual model supports the creation of a database logic model entity and field attributes thereof on a design panel and supports the export of a model from a logic model design interface to be provided for developers to develop data application or import a model created by other tools, wherein the other tools comprise a PowerDesigner enterprise modeling tool; the reverse construction logic model of the database model automatically extracts metadata information of the database table from a source database through the logic model design management module, automatically generates the logic model of the database table, supports manual adjustment and does not need to manually create the database model, wherein the metadata refers to the attribute of the database table; the physical model management module realizes the construction of a physical model, and generates a database execution script DDL statement by using a logical model generated by the logical model design management module as the input of the physical model management module, wherein the metadata of the physical model is changed along with the change of the logical model, the change condition can be tracked according to the version number, and the DDL refers to a database table creation script; the table management module generates an executable database script through the physical model generated by the physical model management module, automatically executes the database script, creates an actual table and realizes the maintenance of the database table.
2. The power grid big data-based warehouse visual modeling system of claim 1, wherein the logic model design management module further comprises a visual data modeling tool module, a visual database modeling tool module, and a reverse engineering module; the visual data modeling tool module is connected with the visual database modeling tool module, and the visual database modeling tool module is connected with the reverse engineering module; and the visual data modeling tool module transmits the change metadata to the visual database modeling module and transmits the large platform data capable of reversely generating the logic model to the reverse engineering module.
3. A modeling method using the grid big data based warehouse visualization modeling system according to claim 2, characterized by comprising the following steps:
s1, a data warehouse layering rule is formulated by a data warehouse engineer according to business requirements and metadata conditions, and the formulated data warehouse layering rule is transmitted to a logic model design management module;
s2, accessing the big data platform data to a data source configuration module to obtain metadata, and transmitting the metadata to a logic model design management module;
s3, generating a data warehouse layering rule and a metadata generation logic model by using the warehouse design in the step S1 and the metadata in the step S2 through a logic model design management module, and transmitting the logic model to a layer/theme domain management module;
s4, the layer/subject domain management module generates a complete power data warehouse logical model of each layer by using the logical model in the step S3, and transmits the complete power data warehouse logical model of each layer to the physical model management module;
s5, the physical model management module generates a physical model by using the complete power data warehouse logical model of each layer in the step S4, and transmits the physical model to the table management module;
s6, the table management module realizes automatic synchronous recovery of data of the data table by using the physical model in the step S5;
and S7, exporting a data model of the designated data, wherein the data model comprises a data structure, data operation and data constraint.
4. The modeling method of the grid big data-based warehouse visualization modeling system according to claim 3, wherein the specific process of step S1 is: and planning an overall data warehouse construction scheme by combining a database table structure in a power grid source database and specifically output data application requirements, and formulating a data warehouse design rule by a data warehouse engineer, wherein the data application comprises power consumption statistical analysis and regional power consumption statistical analysis.
5. The modeling method of the grid big data-based warehouse visualization modeling system according to claim 4, wherein the specific process of the step S2 is: the big data platform is accessed to a required database table to provide metadata for creating a data logic model.
6. The modeling method of the grid big data-based warehouse visualization modeling system according to claim 5, wherein the specific process of the step S3 is:
s3.1: entering a visual data modeling tool module, and realizing the design of a data model through a mode without code dragging; automatically detecting the change of the metadata through the mapping relation between the metadata and the source data structure; metadata change of the data physical model is realized in a version control mode;
s3.2: entering a visual database modeling tool module, and determining whether a logic model is reversely generated through a database table on which a large data platform falls to the ground by a data warehouse construction scheme;
s3.3: extracting database table metadata in a logic model design module;
s3.4: generating a logic model by using a reverse engineering building module in a logic model design module, selecting a database table specified by a large data platform through a patent tool reverse engineering component, building a reverse engineering model of the database table, generating a logic entity and attributes, and automatically building a direct logic relation of the database table according to a main foreign key relation of the database table;
s3.5: and creating a data table logic entity, creating the entity and the attribute thereof in a way of self-defining design of logic test questions, and establishing an association relation with other logic test questions in the logic model.
7. The modeling method of the grid big data-based warehouse visualization modeling system according to claim 6, wherein the specific process of step S4 is: according to a data warehouse layering rule formulated by a data warehouse engineer, establishing a data model of each layer of the data warehouse in an automatic generation mode, and adjusting entity attribute information through a graphical interface; and finally, generating a complete power data warehouse logic model of each level.
8. The modeling method of the grid big data-based warehouse visualization modeling system according to claim 7, wherein the specific process of step S5 is:
s5.1: generating a data physical model, selecting a logic model in a physical model interface, generating a pseudo code script, automatically executing the script and generating the physical model;
s5.2: establishing a mapping relation between metadata and a source library table, establishing a mapping relation between the metadata of a model entity and the source library table, binding a version number, and providing a relational support for automatic change of the data entity or attribute of the model;
s5.3: detecting metadata change, namely automatically detecting whether metadata of a source database table has a change condition in the data acquisition process of a big data platform through the relationship established between the metadata of a model entity and the source database table, if the metadata is detected to have a change, automatically creating a standby data table by a system without influencing data acquisition, automatically completing the change of a data model by the system, and recording the change content and the version;
s5.4: and tracing metadata change, and inquiring metadata change records of each version according to the version number.
9. The modeling method of the grid big data-based warehouse visualization modeling system according to claim 8, wherein the specific process of step S6 is: and changing the physical model entity, automatically generating script codes after changing, realizing physical model change, and realizing automatic synchronous recovery of the standby data table data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011295978.7A CN112579563B (en) | 2020-11-18 | 2020-11-18 | Power grid big data-based warehouse visualization modeling system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011295978.7A CN112579563B (en) | 2020-11-18 | 2020-11-18 | Power grid big data-based warehouse visualization modeling system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112579563A CN112579563A (en) | 2021-03-30 |
| CN112579563B true CN112579563B (en) | 2022-01-21 |
Family
ID=75123002
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011295978.7A Active CN112579563B (en) | 2020-11-18 | 2020-11-18 | Power grid big data-based warehouse visualization modeling system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112579563B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114490578A (en) * | 2022-01-21 | 2022-05-13 | 阿里云计算有限公司 | Data model management method, device and equipment |
| CN114597890A (en) * | 2022-01-27 | 2022-06-07 | 国网冀北电力有限公司经济技术研究院 | Construction method of holographic data system of power transmission line |
| CN116303342B (en) * | 2023-02-01 | 2023-09-12 | 北京三维天地科技股份有限公司 | Reverse model construction method and system |
| CN116303469B (en) * | 2023-03-31 | 2024-01-30 | 佛山众陶联供应链服务有限公司 | Comprehensive and rapid looking-up analysis data warehouse management method and device |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102918530A (en) * | 2010-05-27 | 2013-02-06 | 甲骨文国际公司 | Data mart automation |
| CN103729460A (en) * | 2014-01-10 | 2014-04-16 | 中国南方电网有限责任公司 | Graphical data model managing method and system based on metadata |
| CN105589886A (en) * | 2014-10-24 | 2016-05-18 | 国家电网公司 | Power network public information model construction method and power network public information model construction device |
| CN105787059A (en) * | 2016-02-29 | 2016-07-20 | 四川长虹电器股份有限公司 | Data warehouse based financial data integration method |
| CN107315776A (en) * | 2017-05-27 | 2017-11-03 | 国网安徽省电力公司信息通信分公司 | A kind of data management system based on cloud computing |
| CN109144982A (en) * | 2018-09-29 | 2019-01-04 | 北京友友天宇系统技术有限公司 | Multidimensional holographic Database Dynamic constructing technology system |
| CN109523423A (en) * | 2018-11-28 | 2019-03-26 | 中国海洋石油集团有限公司 | A kind of application system generation method, device, equipment and storage medium |
| CN110046734A (en) * | 2018-12-06 | 2019-07-23 | 广东电网有限责任公司 | Match low pressure power grid grid method for dynamically partitioning and the system of front-end convergence based on battalion |
| CN110168518A (en) * | 2016-11-07 | 2019-08-23 | 塔谱软件公司 | Prepare and arrange the user interface of the data for subsequent analysis |
| CN110489459A (en) * | 2019-08-07 | 2019-11-22 | 国网安徽省电力有限公司 | A kind of enterprise-level industry number fused data analysis system based on big data platform |
| CN111291025A (en) * | 2020-03-10 | 2020-06-16 | 北京东方金信科技有限公司 | Method for supporting multi-physical model conversion by logic model and storage device |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2111593A2 (en) * | 2007-01-26 | 2009-10-28 | Information Resources, Inc. | Analytic platform |
| US20150193519A1 (en) * | 2014-01-09 | 2015-07-09 | International Business Machines Corporation | Modeling and visualizing level-based hierarchies |
| US10902368B2 (en) * | 2014-03-12 | 2021-01-26 | Dt360 Inc. | Intelligent decision synchronization in real time for both discrete and continuous process industries |
| CN105718565B (en) * | 2016-01-20 | 2019-07-02 | 北京京东尚科信息技术有限公司 | The construction method and construction device of data warehouse model |
| CN108520008A (en) * | 2018-03-15 | 2018-09-11 | 链家网（北京）科技有限公司 | The construction method and construction device of data warehouse model |
-
2020
- 2020-11-18 CN CN202011295978.7A patent/CN112579563B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102918530A (en) * | 2010-05-27 | 2013-02-06 | 甲骨文国际公司 | Data mart automation |
| CN103729460A (en) * | 2014-01-10 | 2014-04-16 | 中国南方电网有限责任公司 | Graphical data model managing method and system based on metadata |
| CN105589886A (en) * | 2014-10-24 | 2016-05-18 | 国家电网公司 | Power network public information model construction method and power network public information model construction device |
| CN105787059A (en) * | 2016-02-29 | 2016-07-20 | 四川长虹电器股份有限公司 | Data warehouse based financial data integration method |
| CN110168518A (en) * | 2016-11-07 | 2019-08-23 | 塔谱软件公司 | Prepare and arrange the user interface of the data for subsequent analysis |
| CN107315776A (en) * | 2017-05-27 | 2017-11-03 | 国网安徽省电力公司信息通信分公司 | A kind of data management system based on cloud computing |
| CN109144982A (en) * | 2018-09-29 | 2019-01-04 | 北京友友天宇系统技术有限公司 | Multidimensional holographic Database Dynamic constructing technology system |
| CN109523423A (en) * | 2018-11-28 | 2019-03-26 | 中国海洋石油集团有限公司 | A kind of application system generation method, device, equipment and storage medium |
| CN110046734A (en) * | 2018-12-06 | 2019-07-23 | 广东电网有限责任公司 | Match low pressure power grid grid method for dynamically partitioning and the system of front-end convergence based on battalion |
| CN110489459A (en) * | 2019-08-07 | 2019-11-22 | 国网安徽省电力有限公司 | A kind of enterprise-level industry number fused data analysis system based on big data platform |
| CN111291025A (en) * | 2020-03-10 | 2020-06-16 | 北京东方金信科技有限公司 | Method for supporting multi-physical model conversion by logic model and storage device |
Non-Patent Citations (4)
| Title |
|---|
| Development of a University Financial Data Warehouse and its Visualization Tool;Earl VonF. Lapura等;《Procedia Computer Science》;20180829;第135卷;第587-595页 * |
| PowerDesigner(五)-概念数据模型(CDM生成LDM,PDM和OOM);conviction_thinking;《https://blog.csdn.net/conviction_thinking/article/details/7957367》;20120908;第1页 * |
| 建设数据仓库的八个步骤;leetoclass;《https://zhuanlan.zhihu.com/p/28252482》;20170802;第1页 * |
| 数据仓库实现技术;唐宏等;《数字通信》;20000801(第8期);第12-14页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112579563A (en) | 2021-03-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112579563B (en) | Power grid big data-based warehouse visualization modeling system and method | |
| CN113033001B (en) | Three-dimensional digital factory modeling method and system for digital twin application | |
| CN103020393B (en) | Aircraft assembly data visualization method based on Web | |
| CN111984709A (en) | Visual big data middle station-resource calling and algorithm | |
| CN112182077B (en) | Intelligent operation and maintenance system based on data middling platform technology | |
| CN103825755A (en) | Power secondary system modeling method and system | |
| CN112633822B (en) | Asset management method based on digital twin technology, storage medium and mobile terminal | |
| CN112100800B (en) | Space-time information intelligent analysis system architecture design method oriented to earth edge environment | |
| CN114238488B (en) | Multi-professional digital fruit integration and data interaction method and system | |
| CN110737729A (en) | Engineering map data information management method based on knowledge map concept and technology | |
| CN111666355B (en) | Model construction method and device for field, data and scene three-layer model | |
| CN115617776A (en) | Data management system and method | |
| CN115080275A (en) | Twin service assembly based on real-time data model and method thereof | |
| CN116662441A (en) | Distributed data blood margin construction and display method | |
| CN108875087B (en) | Method for describing object space attribute and searching based on description | |
| CN117076463B (en) | Multi-source data aggregation storage system for smart city | |
| CN113722564A (en) | Visualization method and device for energy and material supply chain based on space map convolution | |
| CN110990907B (en) | Feature-resource knowledge-based three-level optimization method for manufacturability of marine diesel engine heavy parts | |
| CN112784129A (en) | Pump station equipment operation and maintenance data supervision platform | |
| CN117114600A (en) | RPA virtual employee management method and system for power grid production command center | |
| CN111784192A (en) | Industrial park emergency plan executable system based on dynamic evolution | |
| CN109522292B (en) | Data processing device and method based on power grid standard unified information model | |
| CN116842092A (en) | Method and system for database construction and collection management | |
| CN111414355A (en) | Offshore wind farm data monitoring and storing system, method and device | |
| CN115884235A (en) | 5G network digital twin modeling method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |