CN107025285A - A kind of data handling system of comprehensive operation - Google Patents
A kind of data handling system of comprehensive operation Download PDFInfo
- Publication number
- CN107025285A CN107025285A CN201710225319.8A CN201710225319A CN107025285A CN 107025285 A CN107025285 A CN 107025285A CN 201710225319 A CN201710225319 A CN 201710225319A CN 107025285 A CN107025285 A CN 107025285A
- Authority
- CN
- China
- Prior art keywords
- data
- handling system
- comprehensive operation
- model
- service module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0637—Strategic management or analysis, e.g. setting a goal or target of an organisation; Planning actions based on goals; Analysis or evaluation of effectiveness of goals
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Business, Economics & Management (AREA)
- Human Resources & Organizations (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Entrepreneurship & Innovation (AREA)
- Educational Administration (AREA)
- Economics (AREA)
- Strategic Management (AREA)
- Development Economics (AREA)
- Computational Linguistics (AREA)
- Game Theory and Decision Science (AREA)
- Marketing (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The present invention discloses a kind of data handling system of comprehensive operation, including user service module：It is the window of interactive information between user and system for the data and the data of display system output of input；Intermediate process module：Data statistics, form, data mining, comprehensive analysis and the data modeling inputted to user service module；Data service module, the data for storing intermediate process module generation；And online process module, for being managed concentratedly to the data in data service module, key index monitoring and ETL processing；The data handling system of the comprehensive operation can effectively analyze information, so as to help manager to make correct decision-making in time to change in future.
Description
Technical field
The present invention relates to a kind of data handling system of comprehensive operation.
Background technology
With the propulsion and the continuous improvement of application level of IT application in enterprises, the data scale accumulated in enterprise is more and more huger
Greatly.How information is effectively analyzed, so as to help manager to make correct decision-making in time to change in future, finally swashed
It is active in strong market competition, has become the problem of current enterprise more and more urgently is wanted to solve.
The content of the invention
Information can be effectively analyzed the technical problem to be solved in the present invention is to provide one kind, so as to help manager can
The data handling system of the comprehensive operation of correct decision-making in time is made to change in future.
To solve the above problems, the present invention is adopted the following technical scheme that：
A kind of data handling system of comprehensive operation, including
User service module：It is interactive information between user and system for the data and the data of display system output of input
Window；
Intermediate process module：Data statistics, form, data mining, comprehensive analysis and the data that user service module is inputted are built
Mould；
Data service module, the data for storing intermediate process module generation；And
Online process module, for being managed concentratedly to the data in data service module, key index monitoring and ETL at
Reason.
Preferably, the data service module is data warehouse.
Preferably, the data warehouse is the data warehouse of hub-and-spoke configuration.
Preferably, the data handling system of the comprehensive operation also include can according to the default of online process module,
The automatic warning module that alarm is carried out to crucial level of control.
Preferably, the ETL processing comprises the following steps：
1) processing empty value：Field null value is captured, other implication data are loaded or replaced with, and realize according to field null value
Shunting is loaded into different target storehouse.
2）Standardization data format：To field format constraint definition, for the data such as time, numerical value, character in data source
Self-defined load format.
2) data are split：Field is decomposed according to business demand.
3) data correctness is verified：Data verification is carried out with Lookup and fractionation function.
4）Data are replaced：For because of business factor, invalid data, the replacement of missing data can be achieved.
5) Lookup discovers and seizes loss data：Subquery is realized, and returns to the absent field obtained with other means, it is ensured that
Field integrality.
Preferably, the data modeling comprises the following steps:
Step 1, the business event that user service module is submitted is described, retrieval conversion is carried out by word segmentation processing result, set up just
Beginning disassembles form；
Step 2, according to it is initial disassemble form and draw disassemble list；
Step 3, to the list of disassembling of submission, progress includes the processing procedure of preliminary treatment, forms model treatment form；
Step 4, the processing of neutral net is carried out to model treatment form；
Step 5, to the model treatment form after processing, carry out including the processing procedure of output processing, form the table of model definition
Lattice are exported；
Step 6, the model of output is evaluated.
Preferably, the preliminary treatment includes step in detailed below：
Ⅰ）Set up column and include model name, modeling moment, Property Name, reference attribute, correlation model, reference field, father's category
The model treatment form of property；
Ⅱ）For the often capable model name of the model treatment form, modeling moment, Property Name, reference attribute, association mould
Type, reference field, the curriculum offering of parent attribute column are the corresponding column with confirming the same column that form is disassembled described in submission
Content.
Preferably, the neutral net is reverse neural network.
Beneficial effects of the present invention are：By being managed concentratedly to the data in data service module, key index prison
Control and ETL processing, enable analysis personnel rapid, unanimously, alternatively the mode of objective understanding reflects enterprise in all its bearings
Truth.The most of strategy of on-line analytical processing is all that relationship type or common data are carried out into multidimensional data storage, with
It is convenient for analysis.
Brief description of the drawings
Technical scheme in order to illustrate the embodiments of the present invention more clearly, makes required in being described below to embodiment
Accompanying drawing is briefly described, it should be apparent that, drawings in the following description are only some embodiments of the present invention, for
For those of ordinary skill in the art, on the premise of not paying creative work, other can also be obtained according to these accompanying drawings
Accompanying drawing.
Fig. 1 is a kind of connection block diagram of the data handling system of comprehensive operation of embodiment 1；
Fig. 2 is a kind of connection block diagram of the data handling system of comprehensive operation of embodiment 2.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clear, complete
Site preparation is described, it is clear that described embodiment is only a part of embodiment of the invention, rather than whole embodiments.
Embodiment 1
As shown in figure 1, a kind of data handling system of comprehensive operation, including
User service module 1：It is interactive information between user and system for the data and the data of display system output of input
Window；
Intermediate process module 2：Data statistics, form, data mining, comprehensive analysis and the data that user service module is inputted are built
Mould；
Data warehouse 3, the data for storing intermediate process module generation；And
Online process module 4, for being managed concentratedly to the data in data service module, key index monitoring and ETL at
Reason.
The ETL processing comprises the following steps：
1) processing empty value：Field null value is captured, other implication data are loaded or replaced with, and realize according to field null value
Shunting is loaded into different target storehouse.
2）Standardization data format：To field format constraint definition, for the data such as time, numerical value, character in data source
Self-defined load format.
2) data are split：Field is decomposed according to business demand.
3) data correctness is verified：Data verification is carried out with Lookup and fractionation function.
4）Data are replaced：For because of business factor, invalid data, the replacement of missing data can be achieved.
5) Lookup discovers and seizes loss data：Subquery is realized, and returns to the absent field obtained with other means, it is ensured that
Field integrality.
The data modeling comprises the following steps:
Step 1, the business event that user service module is submitted is described, retrieval conversion is carried out by word segmentation processing result, set up just
Beginning disassembles form；
Step 2, according to it is initial disassemble form and draw disassemble list；
Step 3, to the list of disassembling of submission, progress includes the processing procedure of preliminary treatment, forms model treatment form；
Step 4, the processing of neutral net is carried out to model treatment form；
Step 5, to the model treatment form after processing, carry out including the processing procedure of output processing, form the table of model definition
Lattice are exported；
Step 6, the model of output is evaluated.
Beneficial effects of the present invention are：By being managed concentratedly to the data in data service module, key index prison
Control and ETL processing, enable analysis personnel rapid, unanimously, alternatively the mode of objective understanding reflects enterprise in all its bearings
Truth.The most of strategy of on-line analytical processing is all that relationship type or common data are carried out into multidimensional data storage, with
It is convenient for analysis.
Embodiment 2
As shown in Fig. 2 a kind of data handling system of comprehensive operation, including
User service module 1：It is interactive information between user and system for the data and the data of display system output of input
Window；
Intermediate process module 2：Data statistics, form, data mining, comprehensive analysis and the data that user service module is inputted are built
Mould；
The data warehouse 6 of hub-and-spoke configuration, the data for storing intermediate process module generation；
Online process module 4, for being managed concentratedly to the data in data service module, key index monitoring and ETL at
Reason；And
Warning module 5, can carry out alarm to crucial level of control automatically according to the default of online process module.
The ETL processing comprises the following steps：
1) processing empty value：Field null value is captured, other implication data are loaded or replaced with, and realize according to field null value
Shunting is loaded into different target storehouse.
2）Standardization data format：To field format constraint definition, for the data such as time, numerical value, character in data source
Self-defined load format.
2) data are split：Field is decomposed according to business demand.
3) data correctness is verified：Data verification is carried out with Lookup and fractionation function.
4）Data are replaced：For because of business factor, invalid data, the replacement of missing data can be achieved.
5) Lookup discovers and seizes loss data：Subquery is realized, and returns to the absent field obtained with other means, it is ensured that
Field integrality.
The data modeling comprises the following steps:
Step 1, the business event that user service module is submitted is described, retrieval conversion is carried out by word segmentation processing result, set up just
Beginning disassembles form；
Step 2, according to it is initial disassemble form and draw disassemble list；
Step 3, to the list of disassembling of submission, progress includes the processing procedure of preliminary treatment, forms model treatment form；
Step 4, the processing of reverse neural network is carried out to model treatment form；
Step 5, to the model treatment form after processing, carry out including the processing procedure of output processing, form the table of model definition
Lattice are exported；
Step 6, the model of output is evaluated.
The preliminary treatment includes step in detailed below：
Ⅰ）Set up column and include model name, modeling moment, Property Name, reference attribute, correlation model, reference field, father's category
The model treatment form of property；
Ⅱ）For the often capable model name of the model treatment form, modeling moment, Property Name, reference attribute, association mould
Type, reference field, the curriculum offering of parent attribute column are the corresponding column with confirming the same column that form is disassembled described in submission
Content.
Beneficial effects of the present invention are：By being managed concentratedly to the data in data service module, key index prison
Control and ETL processing, enable analysis personnel rapid, unanimously, alternatively the mode of objective understanding reflects enterprise in all its bearings
Truth.The most of strategy of on-line analytical processing is all that relationship type or common data are carried out into multidimensional data storage, with
Analysis is convenient for, and employs the fact that the data warehouse storage of hub-and-spoke configuration be'ss on enterprise data, Star Model is being carried out
During multidimensional data analysis, speed is fast, it is possible to reduce the time required for analysis.
The foregoing is only a specific embodiment of the invention, but protection scope of the present invention is not limited thereto, any
The change or replacement expected without creative work, should all be included within the scope of the present invention.
Claims (8)
1. a kind of data handling system of comprehensive operation, it is characterised in that：Including
User service module：It is interactive information between user and system for the data and the data of display system output of input
Window；
Intermediate process module：Data statistics, form, data mining, comprehensive analysis and the data that user service module is inputted are built
Mould；
Data service module, the data for storing intermediate process module generation；And
Online process module, for being managed concentratedly to the data in data service module, key index monitoring and ETL at
Reason.
2. a kind of data handling system of comprehensive operation according to claim 1, it is characterised in that：The data, services mould
Block is data warehouse.
3. a kind of data handling system of comprehensive operation according to claim 1, it is characterised in that：The data warehouse is
The data warehouse of hub-and-spoke configuration.
4. a kind of data handling system of comprehensive operation according to claim 1, it is characterised in that：Also including can basis
Presetting for online process module, carries out the warning module of alarm to crucial level of control automatically.
5. a kind of data handling system of comprehensive operation according to claim 1, it is characterised in that：The ETL processing bag
Include following steps：
1) processing empty value：Field null value is captured, other implication data are loaded or replaced with, and realize according to field null value
Shunting is loaded into different target storehouse；
2）Standardization data format：To field format constraint definition, made by oneself for the data such as time, numerical value, character in data source
Adopted load format；
2) data are split：Field is decomposed according to business demand；
3) data correctness is verified：Data verification is carried out with Lookup and fractionation function；
4）Data are replaced：For because of business factor, invalid data, the replacement of missing data can be achieved；
5) Lookup discovers and seizes loss data：Subquery is realized, and returns to the absent field obtained with other means, it is ensured that field
Integrality.
6. a kind of data handling system of comprehensive operation according to claim 5, it is characterised in that：The data modeling bag
Include following steps:
Step 1, the business event that user service module is submitted is described, retrieval conversion is carried out by word segmentation processing result, set up just
Beginning disassembles form；
Step 2, according to it is initial disassemble form and draw disassemble list；
Step 3, to the list of disassembling of submission, progress includes the processing procedure of preliminary treatment, forms model treatment form；
Step 4, the processing of neutral net is carried out to model treatment form；
Step 5, to the model treatment form after processing, carry out including the processing procedure of output processing, form the table of model definition
Lattice are exported；
Step 6, the model of output is evaluated.
7. a kind of data handling system of comprehensive operation according to claim 6, it is characterised in that：The preliminary treatment bag
Include step in detailed below：
Ⅰ）Set up column and include model name, modeling moment, Property Name, reference attribute, correlation model, reference field, father's category
The model treatment form of property；
Ⅱ）For the often capable model name of the model treatment form, modeling moment, Property Name, reference attribute, association mould
Type, reference field, the curriculum offering of parent attribute column are the corresponding column with confirming the same column that form is disassembled described in submission
Content.
8. a kind of data handling system of comprehensive operation according to claim 6, it is characterised in that：The neutral net is
Reverse neural network.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710225319.8A CN107025285A (en) | 2017-04-07 | 2017-04-07 | A kind of data handling system of comprehensive operation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710225319.8A CN107025285A (en) | 2017-04-07 | 2017-04-07 | A kind of data handling system of comprehensive operation |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107025285A true CN107025285A (en) | 2017-08-08 |
Family
ID=59527286
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710225319.8A Pending CN107025285A (en) | 2017-04-07 | 2017-04-07 | A kind of data handling system of comprehensive operation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107025285A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108053136A (en) * | 2017-12-27 | 2018-05-18 | 上海储翔信息科技有限公司 | A kind of car dealer's analysis on Achievements expert system based on neutral net |
| CN109063964A (en) * | 2018-07-02 | 2018-12-21 | 浙江百先得服饰有限公司 | A kind of platform data processing system |
| CN109639910A (en) * | 2018-10-19 | 2019-04-16 | 平安科技（深圳）有限公司 | Data interactive method, equipment, storage medium and device |
| CN110930195A (en) * | 2019-11-26 | 2020-03-27 | 中信银行股份有限公司 | Data processing method and electronic equipment |
| CN111159161A (en) * | 2019-12-31 | 2020-05-15 | 青梧桐有限责任公司 | ETL rule-based data quality monitoring and early warning system and method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130166498A1 (en) * | 2011-12-25 | 2013-06-27 | Microsoft Corporation | Model Based OLAP Cube Framework |
| CN105303455A (en) * | 2015-10-16 | 2016-02-03 | 国家电网公司 | Power enterprise user data storage and analysis system |
| CN105426394A (en) * | 2015-10-18 | 2016-03-23 | 广州赛意信息科技股份有限公司 | Mobile report generation method and system based on cross-platform |
| CN105718473A (en) * | 2014-12-05 | 2016-06-29 | 成都复晓科技有限公司 | Data modeling method |
-
2017
- 2017-04-07 CN CN201710225319.8A patent/CN107025285A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130166498A1 (en) * | 2011-12-25 | 2013-06-27 | Microsoft Corporation | Model Based OLAP Cube Framework |
| CN105718473A (en) * | 2014-12-05 | 2016-06-29 | 成都复晓科技有限公司 | Data modeling method |
| CN105303455A (en) * | 2015-10-16 | 2016-02-03 | 国家电网公司 | Power enterprise user data storage and analysis system |
| CN105426394A (en) * | 2015-10-18 | 2016-03-23 | 广州赛意信息科技股份有限公司 | Mobile report generation method and system based on cross-platform |
Non-Patent Citations (1)
| Title |
|---|
| 侯杰等: "基于OLAP的矿业集团生产运营决策系统构建研究", 《中国矿业》 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108053136A (en) * | 2017-12-27 | 2018-05-18 | 上海储翔信息科技有限公司 | A kind of car dealer's analysis on Achievements expert system based on neutral net |
| CN109063964A (en) * | 2018-07-02 | 2018-12-21 | 浙江百先得服饰有限公司 | A kind of platform data processing system |
| CN109639910A (en) * | 2018-10-19 | 2019-04-16 | 平安科技（深圳）有限公司 | Data interactive method, equipment, storage medium and device |
| CN109639910B (en) * | 2018-10-19 | 2021-12-24 | 平安科技（深圳）有限公司 | Data interaction method, equipment, storage medium and device |
| CN110930195A (en) * | 2019-11-26 | 2020-03-27 | 中信银行股份有限公司 | Data processing method and electronic equipment |
| CN111159161A (en) * | 2019-12-31 | 2020-05-15 | 青梧桐有限责任公司 | ETL rule-based data quality monitoring and early warning system and method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107025285A (en) | A kind of data handling system of comprehensive operation | |
| CN107368967A (en) | Engineering safety quality inspection intelligent management based on internet | |
| CN113553420B (en) | Power grid fault processing rule recommendation method and system based on knowledge graph | |
| JP2565310B2 (en) | Knowledge base to database converter | |
| CN110162644B (en) | Image set establishing method, device and storage medium | |
| CN106980634A (en) | System and method for classifying and solving software production accident list | |
| CN107958337A (en) | A kind of information resources visualize mobile management system | |
| CN104216948B (en) | A kind of difference control method of data model | |
| CN107103448A (en) | Data integrated system based on workflow | |
| CN102708446B (en) | A kind of method of setting up the emergent intelligent command system of flood control | |
| CN109299908A (en) | A kind of remote visualization assistance system | |
| CN115858796A (en) | Fault knowledge graph construction method and device | |
| CN109460393B (en) | Big data-based visual system for pre-inspection and pre-repair | |
| CN116468287A (en) | Intelligent park control system based on digital twinning | |
| CN101710322A (en) | Method and system for associating information | |
| CN107301245A (en) | A kind of power information video searching system | |
| CN110007905A (en) | A kind of generation method and system of the software development scheme based on big data | |
| CN106779535A (en) | A kind of product warehousing and ex-warehouse address changing method and system | |
| CN114003600A (en) | Data processing method, system, electronic device and storage medium | |
| US10754861B2 (en) | System and method for content affinity analytics | |
| CN109033330A (en) | Big data cleaning method, device and server | |
| CN113434627A (en) | Work order processing method and device and computer readable storage medium | |
| CN116821423A (en) | Intelligent analysis and knowledge type fault processing auxiliary system and method for power distribution network | |
| CN107451283A (en) | The analysis method and device of data | |
| Gao et al. | Research of Internet of Things applied to manufacturing execution system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170808 |