CN115794804A - Engineering internal control data visualization processing system and method based on big data technology - Google Patents
Engineering internal control data visualization processing system and method based on big data technology Download PDFInfo
- Publication number
- CN115794804A CN115794804A CN202310072094.2A CN202310072094A CN115794804A CN 115794804 A CN115794804 A CN 115794804A CN 202310072094 A CN202310072094 A CN 202310072094A CN 115794804 A CN115794804 A CN 115794804A
- Authority
- CN
- China
- Prior art keywords
- data
- analysis
- statistical analysis
- layer
- internal control
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 55
- 238000005516 engineering process Methods 0.000 title claims abstract description 26
- 238000000034 method Methods 0.000 title claims description 22
- 238000013079 data visualisation Methods 0.000 title claims description 19
- 238000004458 analytical method Methods 0.000 claims abstract description 47
- 238000007619 statistical method Methods 0.000 claims abstract description 40
- 238000012800 visualization Methods 0.000 claims abstract description 22
- 230000000007 visual effect Effects 0.000 claims abstract description 18
- 238000013439 planning Methods 0.000 claims abstract description 13
- 238000003672 processing method Methods 0.000 claims abstract description 12
- 238000006243 chemical reaction Methods 0.000 claims abstract description 8
- 238000005192 partition Methods 0.000 claims description 19
- 238000003860 storage Methods 0.000 claims description 10
- 238000004140 cleaning Methods 0.000 claims description 9
- 238000012544 monitoring process Methods 0.000 claims description 9
- 238000013075 data extraction Methods 0.000 claims description 8
- 230000008569 process Effects 0.000 claims description 7
- 238000013179 statistical model Methods 0.000 claims description 6
- 230000002776 aggregation Effects 0.000 claims description 5
- 238000004220 aggregation Methods 0.000 claims description 5
- 238000006116 polymerization reaction Methods 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 4
- 238000010276 construction Methods 0.000 claims description 4
- 238000007728 cost analysis Methods 0.000 claims description 4
- 230000036541 health Effects 0.000 claims description 4
- 238000013475 authorization Methods 0.000 claims description 3
- 230000005540 biological transmission Effects 0.000 claims description 3
- 210000001503 joint Anatomy 0.000 claims description 3
- 230000002596 correlated effect Effects 0.000 abstract description 4
- 238000007726 management method Methods 0.000 description 8
- 230000000875 corresponding effect Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000007405 data analysis Methods 0.000 description 2
- 238000013479 data entry Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000004931 aggregating effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a visual processing system and a visual processing method for engineering internal control data based on a big data technology, wherein the visual processing system comprises: the business system is used for providing a database interface and transmitting data to the data warehouse; the data warehouse is used for performing hierarchical planning on the data acquired from the business system, performing standardized conversion, statistics and analysis processing on the data to form statistical analysis data, and pushing the statistical analysis data to the relational database; and the visualization tool is used for butting the relational database and graphically displaying the statistical analysis data. According to the invention, through extracting the data of each service system, further performing statistics and analysis processing, finally generating statistical analysis data and pushing the statistical analysis data to the relational database for graphical display by a visualization tool, the technical problems that the data formats of the systems are not uniform, the entry, processing and display of the data of the systems are relatively independent, the data of the systems cannot be effectively correlated, and the transverse comparison of the data is difficult are solved.
Description
Technical Field
The invention relates to a large data technology-based visual processing system and method for engineering internal control data, and belongs to the technical field of data processing.
Background
At present, data such as power grid enterprise engineering, materials, human resources and financial affairs are scattered in a plurality of systems, data formats among the systems are not uniform, the data entry, processing and display of the systems are relatively independent, the data of the systems cannot be effectively correlated, and the transverse comparison of the data is difficult. The main problems include:
related business systems related to engineering, such as material management, human resource management and control, financial management and control, NC systems and other related systems, have relatively independent data, relatively independent business data storage among the systems, and incapability of performing related query among the systems, so that the data are difficult to transversely compare;
the data magnitude related to the related service system is large, the daily output data volume is more than ten million, the traditional relational database cannot support high-efficiency query of statistical data, and the performance bottleneck exists in the query and statistics of the data;
the statistical billboard based on the whole process of the engineering data can not be realized, and comprises nodes such as the acquisition condition of a project, process management, cost management, management results, archive management and the like;
the data statistics has the timeliness problem, at present, the data statistics of related indexes needs to be carried out by exporting all system data into a table and combining and calculating a plurality of tables under lines, so that the working efficiency is low, and the data timeliness is poor;
statistical data is not carried by a flexible BI tool, and a user-configurable BI tool needs to be provided for flexible display of data.
Therefore, a visualization processing system and method based on big data technology are needed to solve the above technical problems.
Disclosure of Invention
The invention provides a large data technology-based visual processing system and method for engineering internal control data, which are used for solving the technical problems that data formats of systems are not unified, the input, processing and display of the system data are relatively independent, the system data cannot be effectively correlated, and the transverse comparison of the data is difficult. The specific technical scheme is as follows:
engineering internal control data visualization processing system based on big data technology includes:
the business system is used for providing a database interface and transmitting data to the data warehouse;
the data warehouse is used for performing hierarchical planning on data acquired from the business system, performing standardized conversion, statistics and analysis processing on the data to form statistical analysis data, and pushing the statistical analysis data to the relational database;
and the visualization tool is used for butting the relational database and graphically displaying the statistical analysis data.
Preferably, the specific planning hierarchy used by the data warehouse to perform hierarchical planning on the data includes:
the source pasting layer is used for copying the source data of the service system one by one, providing original data support for the sharing layer and archiving the original data of the service system;
the sharing layer is used for carrying out standardized conversion on the extracted service system data, namely cleaning and processing the source layer data, carrying out mild aggregation at the same time, and removing null data, dirty data and outliers to form data wide table storage;
and the analysis layer is used for carrying out statistics and analysis processing on the data processed by the sharing layer to form statistical analysis data which is consistent with the data structure displayed by the visual tool and pushing the statistical analysis data to the relational database.
Preferably, the statistical analysis data includes:
trend data provides data support for operation analysis, including bid-winning condition analysis, underwriting project analysis, income analysis, profit analysis and operation cost analysis;
summarizing data, and providing data support for engineering health degree, wherein the data support comprises scale property, value-added property, efficiency, quality, cost, safety and adaptability;
and the doubtful data provides data support for monitoring and early warning, and comprises key index monitoring, contract signing conditions and contract performance conditions in the whole process of the engineering, and also comprises the conditions of unmatched income cost, excessive project cost, excessive subpackage proportion, untimely contract signing and inconsistent construction contents.
Preferably, the data warehouse further comprises:
the data extraction module is used for acquiring source data from the service system;
and the data pushing module is used for pushing the statistical analysis data formed by the data warehouse to the relational database.
Preferably, the data access mode of the service system and the data warehouse for data transmission can be accessed through an API interface provided by the service system, or transmitted through an ActiveMQ message queue and a Kafka message queue.
Furthermore, the source end data acquired from the business system is subjected to authority division by the source layer, authorization is carried out according to a use department, and corresponding department roles and personnel authorities are established; the data acquisition mode of the source layer is a full acquisition mode, a time partition field is added to the data, the service dimension of the time partition is in a year-month partition mode or a year-month-day partition mode, and the partition data is reserved for more than half a year based on the time partition field.
Preferably, the visualization tool constructs a project statistical model for data presentation.
Furthermore, the project statistical model can be used as basic data for user-defined BI display, and is used for secondary processing and flexible display by a user.
The visualization processing method of the engineering internal control data visualization processing system based on the big data technology comprises the following steps:
step 1: the data extraction module of the data warehouse is connected with a database interface provided by the service system, and the data of the source end of the service system is extracted;
and 2, step: storing source end data extracted from a business system to a source pasting layer of a data warehouse according to an original data structure;
and 3, step 3: cleaning and processing the data of the source pasting layer through a sharing layer of a data warehouse, and simultaneously carrying out slight polymerization to form a data wide table for storage;
and 4, step 4: carrying out statistics and analysis processing on the data of the sharing layer through an analysis layer of the data warehouse to form statistical analysis data which is consistent with a data structure displayed by a visual tool, and pushing the statistical analysis data to a relational database;
and 5: and (4) using a visualization tool to butt joint the relational database in the step (4) and graphically displaying the statistical analysis data.
An electronic device comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein the processor executes the program to realize a visualization processing method of an engineering internal control data visualization processing system based on big data technology.
The method carries out hierarchical planning on the data of the data warehouse, realizes one-to-one copying on the source end data of each service system through the source layer pasting, and reduces the influence of data statistics on the original service system; cleaning and processing the data of the source pasting layer through the sharing layer, and simultaneously carrying out slight polymerization to form a data wide table for storage; data sharing among all service departments is realized through the analysis layer, cross-service data correlation query can be carried out, final statistical analysis data are stored, and the statistical analysis data can be pushed to a relational database one by one for visual display. According to the invention, the data of each service system is extracted through the data warehouse, and is further subjected to statistics and analysis processing, and finally statistical analysis data is generated and is pushed to the relational database for graphical display by a visualization tool, so that the technical problems that the data formats of the systems are not uniform, the data entry, processing and display of the systems are relatively independent, the data of the systems cannot be effectively correlated, and the transverse comparison of the data is difficult are solved.
Drawings
FIG. 1 is a system structure diagram of an engineering internal control data visualization processing system based on big data technology.
FIG. 2 is a flow chart of a method of the engineering internal control data visualization processing method based on big data technology.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is further described in detail below with reference to the accompanying drawings in combination with specific embodiments.
It should be noted that technical terms or scientific terms used in the embodiments of the present application should have a general meaning as understood by those having ordinary skill in the art to which the present application belongs, unless otherwise defined. The use of "first," "second," and similar terms in the embodiments of the present application is not intended to indicate any order, quantity, or importance, but rather is used to distinguish one element from another. The word "comprising" or "comprises", and the like, means that the element or item listed before the word covers the element or item listed after the word and its equivalents, but does not exclude other elements or items. The terms "connected" or "coupled" and the like are not restricted to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "upper", "lower", "left", "right", and the like are used merely to indicate relative positional relationships, and when the absolute position of the object being described is changed, the relative positional relationships may also be changed accordingly.
The data of all business systems are integrated in a unified mode, all the business systems are extracted to a data warehouse, the data are planned in the data warehouse in a layered mode, the planning comprises a source pasting layer ODS, a sharing layer DW and an analysis layer ADS, the source pasting layer mainly achieves one-to-one copying of source end data of all the business systems, influences of data statistics on original business systems are reduced, original data support is provided for a sharing layer, the sharing layer mainly cleans and normalizes the source pasting layer data, null data, dirty data, outliers and the like are removed, meanwhile, light aggregation statistics can be conducted on the data to form a wide table which is used for providing business query, OLAP analysis, data distribution and the like for the analysis layer, the data analysis layer mainly provides data support for data product and data analysis and mainly stores final report statistical data, cross-business data association query can be conducted on the layer, data of all departments are shared mutually, and the data of the layer can be pushed to a relational database one-to-carry out visual display. The specific technical scheme is as follows:
as shown in fig. 1, the system for visualizing and processing engineering internal control data based on big data technology comprises:
and the data extraction module is used for acquiring the source data from the service system.
And the business system is used for providing a database interface and transmitting data to the data warehouse. The data access mode of the data transmission between the service system and the data warehouse can be accessed through an API (application program interface) provided by the service system or transmitted through message queues such as ActiveMQ and Kafka.
The data warehouse is used for performing hierarchical planning on data acquired from the business system, performing standardized conversion, statistics and analysis processing on the data to form statistical analysis data, and pushing the statistical analysis data to the relational database;
the specific planning hierarchy used by the data warehouse to perform hierarchical planning on data includes:
the source pasting layer is used for copying the data of the source end of the service system one by one, providing original data support for the sharing layer and archiving the original data of the service system;
the sharing layer is used for carrying out standardized conversion on the extracted service system data, namely cleaning and processing the source layer data, carrying out mild aggregation at the same time, and removing null data, dirty data and outliers to form data wide table storage;
and the analysis layer is used for carrying out statistics and analysis processing on the data processed by the sharing layer to form statistical analysis data which is consistent with the data structure displayed by the visual tool and pushing the statistical analysis data to the relational database.
And the data pushing module is used for pushing the statistical analysis data formed by the data warehouse to the relational database.
And the visualization tool is used for butting the relational database and graphically displaying the statistical analysis data.
Further, the statistical analysis data includes:
trend data, which provides data support for operation analysis, including bid-winning condition analysis, solicited project analysis, income analysis, profit analysis and operation cost analysis;
summarizing data, and providing data support for engineering health degree, wherein the data support comprises scale property, value-added property, efficiency, quality, cost, safety and adaptability;
and the doubtful data provides data support for monitoring and early warning, and comprises key index monitoring, contract signing conditions and contract performance conditions in the whole process of the engineering, and also comprises the conditions of unmatched income cost, excessive project cost, excessive subpackage proportion, untimely contract signing and inconsistent construction contents.
Furthermore, the source data acquired from the business system is subjected to authority division by the source layer, authorization is carried out according to the service system using condition and using departments (such as a material department and a personnel department), corresponding department roles and personnel authorities are established, and a data sheet is authorized to related department roles; the data acquisition mode of the source layer is a full acquisition mode, a time partition field is added to the data, the service dimension of the time partition is in a year-month partition mode or a year-month-day partition mode, the partition data is reserved for more than half a year by taking the time partition field as a standard, and the traceability of the data is guaranteed.
Further, the visualization tool constructs a project statistical model for data display. The project statistical model can be used as basic data for custom BI display, and a user can carry out secondary processing and flexibly display the data.
As shown in fig. 2, the visualization processing method of the engineering internal control data visualization processing system based on the big data technology includes the following steps:
step 1: the data extraction module of the data warehouse is connected with a database interface provided by the service system, and the data of the source end of the service system is extracted;
step 2: storing source end data extracted from a business system to a source pasting layer of a data warehouse according to an original data structure;
and step 3: cleaning and processing the source pasting layer data through a sharing layer of a data warehouse, and simultaneously performing slight polymerization to form a data wide table for storage;
and 4, step 4: carrying out statistics and analysis processing on the data of the sharing layer through an analysis layer of the data warehouse to form statistical analysis data which is consistent with a data structure displayed by a visual tool, and pushing the statistical analysis data to a relational database;
and 5: and (4) using a visualization tool to butt joint the relational database in the step (4) and graphically displaying the statistical analysis data.
It should be noted that, in step 3, only the data table or the data field concerned may be cleaned and processed according to the service requirement, for example, the log information table, the information of the maintenance field such as the generation time and the update time may be ignored.
The method carries out hierarchical planning on the data of the data warehouse, realizes one-to-one copying on the source end data of each service system through the source layer pasting, and reduces the influence of data statistics on the original service system; cleaning and processing the data of the source pasting layer through the sharing layer, and simultaneously performing slight polymerization to form a data wide table for storage; the data sharing among all the service departments is realized through the analysis layer, cross-service data correlation query can be carried out, the final statistical analysis data are stored, and the statistical analysis data can be pushed to the relational database one by one for visual display. According to the data processing method and system, data of all service systems are extracted through the data warehouse, statistics and analysis processing are further carried out, statistics and analysis data are finally generated and pushed to the relational database for a visualization tool to carry out graphical display, correlation query can be carried out among all systems with relatively independent data storage, and transverse comparison of the data is facilitated. The data warehouse has large storage capacity and can support the efficient query of statistical data. The invention has high working efficiency, can ensure the timeliness of data, does not need to lead out tables by each business system, and completes the statistical work by combining and calculating a plurality of tables under lines. The statistical billboard based on the whole process of engineering data is realized through a visualization tool, and the display is more visual.
The engineering internal control visualization processing method based on the big data technology in one embodiment of the invention comprises the following steps:
step 1: the method comprises the following steps of extracting data from a database of a material, human and financial business system related to engineering, and storing the data in a source pasting layer of a data warehouse, wherein the method specifically comprises the following steps: and creating a data extraction task by using the DataX, and storing the table related to each service system data to a posting layer in the data warehouse.
And the posting layer is used for archiving the original data of all the business systems, establishing roles of engineering, materials, human resources, finance and the like in a data warehouse according to the data source of the extracted business system, and newly adding management users under the corresponding roles.
And newly adding a partition field on a data table of the source layer, mainly dividing monthly data and daily data according to the data extraction frequency, carrying out full-volume acquisition each time by dividing the monthly data according to the month, and dividing the daily data according to the day, and acquiring incremental data each time.
Step 2: in a sharing layer of a data warehouse, the extracted business system data is subjected to standardized conversion to form data wide table storage, and the specific method comprises the following steps:
creating a hive data processing and scheduling task, cleaning and slightly aggregating data, and removing empty fields and irrelevant service fields in the table according to the database design description. And performing association query or aggregation query on the related tables of the same service, taking the query results as a result table to be unified, performing joint query on a plurality of associated tables, naming the result table by using dws as a prefix, and storing the result table.
And step 3: in an analysis layer of a data warehouse, carrying out statistics and analysis processing on the data of the sharing layer to form trend data, summarized data and suspicious data, and storing the statistical data for final display, wherein the statistical data comprises three categories of trend data, summarized data and suspicious data:
the trend data is used for providing relevant data support for operation analysis, and mainly comprises bid-winning condition analysis, underwriting project analysis, income analysis, profit analysis and operation cost analysis;
the summarized data provides related data support for the engineering health degree, and mainly comprises the characteristics of scale, value-added property, efficiency, quality, cost, safety and adaptability;
the doubtful point data provides data support for monitoring and early warning, and mainly comprises key index monitoring, contract signing conditions and contract performance conditions in the whole process of engineering, wherein the key index monitoring, the contract signing conditions and the contract performance conditions also comprise the conditions of unmatched income cost, overrated project cost, overrated subpackage proportion, untimely contract signing, inconsistent construction contents and the like.
And 4, step 4: the method comprises the following steps of pushing trend data, summarized data and suspicious data to a relational database MYSQL for visual display, and specifically comprises the following steps:
creating a push scheduling task by using DataX, and pushing all statistical data to relational data plus MYSQL by month, day and week according to a service scene;
on a DataV large-screen data display component platform, large-screen visual interface design is carried out by using components such as charts, line charts, radar charts, instrument panels, maps and the like, and the pattern of the chart is modified through component parameters, wherein the pattern comprises arrangement mode, tone, size, spacing, radian and the like;
and (4) carrying out data association on all chart components by carrying out dynamic chart data.
The specific method for carrying out data association on all chart components comprises the following steps:
the method comprises the steps of supporting static data, CSV data, API data and a database by default through configuring a component data source, configuring link and authentication information of MYSQL (structured query language) of a relational database, designating a table name, and ensuring dynamic display of data through configuring data updating frequency.
An electronic device comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein the processor executes the program to realize a visualization processing method of an engineering internal control data visualization processing system based on big data technology. The processor may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits, and is configured to execute a relevant program to implement the technical solutions provided in the embodiments of the present specification. The Memory may be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static Memory device, a dynamic Memory device, or the like. The memory may store an operating system and other application programs, and when the technical solution provided by the embodiments of the present specification is implemented by software or firmware, the relevant program codes are stored in the memory and called by the processor to be executed.
It should be noted that although the above device only shows a processor and a memory, in a specific implementation, the device may also include other components necessary for normal operation. In addition, those skilled in the art will appreciate that the above-described apparatus may also include only those components necessary to implement the embodiments of the present disclosure. The electronic device of the above embodiment is used to implement the corresponding engineering internal control data visualization processing method based on the big data technology in the foregoing embodiment, and has the beneficial effects of the corresponding method embodiment, which are not described herein again. The computer program is used for enabling the computer to execute the engineering internal control data visualization processing method based on big data technology according to the embodiment.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, is limited to these examples; within the context of the present application, technical features in the above embodiments or in different embodiments may also be combined, steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application described above, which are not provided in detail for the sake of brevity.
In addition, well-known power/ground connections to Integrated Circuit (IC) chips and other components may or may not be shown in the provided figures for simplicity of illustration and discussion, and so as not to obscure the embodiments of the application. Furthermore, devices may be shown in block diagram form in order to avoid obscuring embodiments of the application, and this also takes into account the fact that specifics with respect to implementation of such block diagram devices are highly dependent upon the platform within which the embodiments of the application are to be implemented (i.e., specifics should be well within purview of one skilled in the art). Where specific details (e.g., circuits) are set forth in order to describe example embodiments of the application, it should be apparent to one skilled in the art that the embodiments of the application can be practiced without, or with variation of, these specific details. Accordingly, the description is to be regarded as illustrative instead of restrictive.
While the present application has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art in light of the foregoing description. For example, other memory architectures, such as Dynamic RAM (DRAM), may use the discussed embodiments.
The present embodiments are intended to embrace all such alternatives, modifications and variances which fall within the broad scope of the appended claims. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of the embodiments of the present application are intended to be included within the scope of the present application.
Claims (10)
1. Engineering internal control data visualization processing system based on big data technology, its characterized in that includes:
the business system is used for providing a database interface and transmitting data to the data warehouse;
the data warehouse is used for performing hierarchical planning on the data acquired from the business system, performing standardized conversion, statistics and analysis processing on the data to form statistical analysis data, and pushing the statistical analysis data to the relational database;
and the visualization tool is used for butting the relational database and graphically displaying the statistical analysis data.
2. The big data technology-based engineering internal control data visualization processing system according to claim 1, wherein the specific planning hierarchy used by the data warehouse to perform hierarchical planning on data comprises:
the source pasting layer is used for copying the data of the source end of the service system one by one, providing original data support for the sharing layer and archiving the original data of the service system;
the sharing layer is used for carrying out standardized conversion on the extracted service system data, namely cleaning and processing the source layer data, carrying out mild aggregation at the same time, and removing null data, dirty data and outliers to form data wide table storage;
and the analysis layer is used for carrying out statistics and analysis processing on the data processed by the sharing layer to form statistical analysis data which is consistent with the data structure displayed by the visual tool and pushing the statistical analysis data to the relational database.
3. The big data technology-based engineering internal control data visualization processing system according to claim 1, wherein the statistical analysis data comprises:
trend data provides data support for operation analysis, including bid-winning condition analysis, underwriting project analysis, income analysis, profit analysis and operation cost analysis;
summarizing data, and providing data support for engineering health degree, wherein the data support comprises scale property, value-added property, efficiency, quality, cost, safety and adaptability;
and the doubtful data provides data support for monitoring and early warning, and comprises key index monitoring, contract signing conditions and contract performance conditions in the whole process of the project, and also comprises the conditions of unmatched income cost, excessive project cost, excessive sub-package proportion, untimely contract signing and inconsistent construction content.
4. The big data technology-based engineering internal control data visualization processing system according to claim 1, wherein the data warehouse further comprises:
the data extraction module is used for acquiring source data from the service system;
and the data pushing module is used for pushing the statistical analysis data formed by the data warehouse to the relational database.
5. The engineering internal control data visualization processing system based on big data technology as claimed in claim 1, wherein the data access mode of the business system and the data warehouse for data transmission can be accessed through API interface provided by the business system, or transmitted through ActiveMQ and Kafka message queue.
6. The visual processing system of engineering internal control data based on big data technology as claimed in claim 2, characterized in that said pasting layer performs authority division for source data obtained from a business system, performs authorization according to a use department, and establishes corresponding department role and personnel authority; the data acquisition mode of the source layer is a full acquisition mode, a time partition field is added to the data, the service dimension of the time partition is in a year-month partition mode or a year-month-day partition mode, and the partition data is reserved for more than half a year based on the time partition field.
7. The big data technology-based engineering internal control data visualization processing system according to claim 1, wherein the visualization tool builds a project statistical model for data presentation.
8. The big data technology-based engineering internal control data visualization processing system as claimed in claim 7, wherein the project statistical model can be used as basic data for custom BI presentation for secondary processing and flexible presentation by users.
9. The visualization processing method of the engineering internal control data visualization processing system based on the big data technology according to any one of the claims 1 to 8, characterized by comprising the following steps:
step 1: the data extraction module of the data warehouse is connected with a database interface provided by the service system, and the data of the source end of the service system is extracted;
step 2: storing source end data extracted from a business system to a source pasting layer of a data warehouse according to an original data structure;
and 3, step 3: cleaning and processing the data of the source pasting layer through a sharing layer of a data warehouse, and simultaneously carrying out slight polymerization to form a data wide table for storage;
and 4, step 4: carrying out statistics and analysis processing on the data of the sharing layer through an analysis layer of the data warehouse to form statistical analysis data which is consistent with a data structure displayed by a visual tool, and pushing the statistical analysis data to a relational database;
and 5: and (4) using a visualization tool to butt joint the relational database in the step (4) and graphically displaying the statistical analysis data.
10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the visualization processing method according to claim 9 when executing the program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310072094.2A CN115794804A (en) | 2023-02-07 | 2023-02-07 | Engineering internal control data visualization processing system and method based on big data technology |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310072094.2A CN115794804A (en) | 2023-02-07 | 2023-02-07 | Engineering internal control data visualization processing system and method based on big data technology |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115794804A true CN115794804A (en) | 2023-03-14 |
Family
ID=85430213
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310072094.2A Pending CN115794804A (en) | 2023-02-07 | 2023-02-07 | Engineering internal control data visualization processing system and method based on big data technology |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115794804A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117034125A (en) * | 2023-10-08 | 2023-11-10 | 江苏臻云技术有限公司 | Classification management system and method for big data fusion |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2007072501A2 (en) * | 2005-12-19 | 2007-06-28 | Mphasis Bfl Limited | A system and a methodology for providing integrated business performance management platform |
| CN101075304A (en) * | 2006-05-18 | 2007-11-21 | 河北全通通信有限公司 | Method for constructing decision supporting system of telecommunication industry based on database |
| CN112241543A (en) * | 2020-10-27 | 2021-01-19 | 国网福建省电力有限公司信息通信分公司 | Sensitive data combing method based on data middling stage |
| CN112883001A (en) * | 2021-01-28 | 2021-06-01 | 国网冀北电力有限公司智能配电网中心 | Data processing method, device and medium based on marketing and distribution through data visualization platform |
| CN114218309A (en) * | 2021-11-04 | 2022-03-22 | 招银云创信息技术有限公司 | Data processing method, system and computer equipment |
| CN114490886A (en) * | 2021-12-29 | 2022-05-13 | 北京航天智造科技发展有限公司 | Industrial operation system data lake construction method based on data warehouse |
-
2023
- 2023-02-07 CN CN202310072094.2A patent/CN115794804A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2007072501A2 (en) * | 2005-12-19 | 2007-06-28 | Mphasis Bfl Limited | A system and a methodology for providing integrated business performance management platform |
| CN101075304A (en) * | 2006-05-18 | 2007-11-21 | 河北全通通信有限公司 | Method for constructing decision supporting system of telecommunication industry based on database |
| CN112241543A (en) * | 2020-10-27 | 2021-01-19 | 国网福建省电力有限公司信息通信分公司 | Sensitive data combing method based on data middling stage |
| CN112883001A (en) * | 2021-01-28 | 2021-06-01 | 国网冀北电力有限公司智能配电网中心 | Data processing method, device and medium based on marketing and distribution through data visualization platform |
| CN114218309A (en) * | 2021-11-04 | 2022-03-22 | 招银云创信息技术有限公司 | Data processing method, system and computer equipment |
| CN114490886A (en) * | 2021-12-29 | 2022-05-13 | 北京航天智造科技发展有限公司 | Industrial operation system data lake construction method based on data warehouse |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117034125A (en) * | 2023-10-08 | 2023-11-10 | 江苏臻云技术有限公司 | Classification management system and method for big data fusion |
| CN117034125B (en) * | 2023-10-08 | 2024-01-16 | 江苏臻云技术有限公司 | Classification management system and method for big data fusion |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109241159B (en) | Partition query method and system for data cube and terminal equipment | |
| CN110168529A (en) | Date storage method, device and storage medium | |
| CN102917009B (en) | A kind of stock certificate data collection based on cloud computing technology and storage means and system | |
| CN110765750B (en) | Report data input method and terminal equipment | |
| CN111897831A (en) | Service message generation method and device, electronic equipment and storage medium | |
| CN106021528A (en) | Data processing method and device | |
| CN115794804A (en) | Engineering internal control data visualization processing system and method based on big data technology | |
| CN113220728A (en) | Data query method, device, equipment and storage medium | |
| EP4216076A1 (en) | Method and apparatus of processing an observation information, electronic device and storage medium | |
| JP2024509629A (en) | Quantum car type component basic database creation method, device, electronic equipment, and storage medium | |
| CN114428813A (en) | Data statistics method, device, equipment and storage medium based on report platform | |
| CN115168752A (en) | Big data query method and device, electronic equipment and storage medium | |
| CN115905397A (en) | Visual display method, device, system and medium for business data | |
| CN114860851A (en) | Data processing method, device, equipment and storage medium | |
| CN113722296A (en) | Agricultural information processing method and device, electronic equipment and storage medium | |
| US20190266526A1 (en) | Multi-dimensional organization of data for efficient analysis | |
| CN112817930A (en) | Data migration method and device | |
| CN114218217A (en) | Custom development method, device, equipment and storage medium for report | |
| CN113468173B (en) | Data storage method, device, equipment and storage medium | |
| CN117632486A (en) | Economic census method and device based on edge calculation | |
| CN114281789A (en) | Business report generation method and device | |
| CN116738953A (en) | Report generation method, report generation device, computer equipment and computer readable storage medium | |
| CN113919696A (en) | Project full life cycle early warning method and device based on big building data | |
| CN114358636A (en) | Index configuration method, data acquisition method, device, equipment and medium | |
| CN117390011A (en) | Report data processing method, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20230314 |