CN104504165A - Mass data based comprehensive management analysis system - Google Patents
Mass data based comprehensive management analysis system Download PDFInfo
- Publication number
- CN104504165A CN104504165A CN201510046328.1A CN201510046328A CN104504165A CN 104504165 A CN104504165 A CN 104504165A CN 201510046328 A CN201510046328 A CN 201510046328A CN 104504165 A CN104504165 A CN 104504165A
- Authority
- CN
- China
- Prior art keywords
- server
- mass data
- load
- task
- distributed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention relates to the technical field of data management, in particular to a mass data based comprehensive management analysis system. The mass data based comprehensive management analysis system comprises the internet, a firewall, a core switch, a load balancing server, a switch and a server cluster which are connected in sequence, wherein the server cluster comprises multiple servers, data are scattered and stored on multiple independent servers, the multiple independent servers share responsibility for load storage, a position server is utilized to position stored information, the computation capacity of an idle computer can be fully utilized to divide the problem which can be solved by relaying on huge computation capacity into multiple portions through adaptation of a MapTeduce based computation model and distributed computation, then the multiple portions are allocated to the multiple servers to be processed, and finally computation results are gathered up to obtain a final result. The mass data in the professional field of hot-line works are stored and analyzed to provide a quick, efficient and reliable aided decision-making support means for hot-line work process quality improvement and decision-making support.
Description
Technical field
The present invention relates to technical field of data administration, be specifically related to the integrated management analytic system based on mass data.
Background technology
Due to the professional requirement of hot line job professional work in hot line job management process, often there is a large amount of work datas and training data, in management in the past, can only retain as file these historical datas, cannot well inquire about, let alone carry out excavation and the analysis of data, therefore need by setting up a kind of comprehension management based on mass data and analytic system, to meet the requirement of hot line job specialized management, improve hot line job processing quality and decision-making provides fast, efficient, the means of aid decision making support reliably.
The main the following aspects of system is considered:
(1) hot line job is a professional administrative skill, need, by infosystem, hot line job specialized standard is carried out standardization solidification and management, realize the robotization of hot line job, intelligent management, to improve standardized level and the efficiency of management of hot line job management;
(2) hot line job has higher professional feature, higher to hot line job professional skill set requirements, need to be shared and training intercourse platform by the professional knowledge that Information System configuration is perfect, to help hot line job personnel to improve professional skill, reduce operating risk;
(3) hot line job is the important component part of electrical network production operation, the all kinds of several of the magnanimity produced in operation process have vital role to equipment state evaluate and analyze, need to set up the specialty analysis system analyzed based on mass data, integrated management and analytical applications are carried out to these data;
(4) along with the sharp increase of electrical network scale, increasing hot line job work within the scope of electrical network, traditional management mode standardization is low, poor in timeliness, the needs of hot line job specialized management can not be met, but because it is professional, still lack the special comprehensive management and analysis based on mass data at present;
(5) along with the quickening increasingly of company information pace of construction, informatization requires more and more higher, hot line job branch office is as the newly established branch office of grid company, be zero substantially in informatization, in order to satisfied informationization application covering waits requirement, in the urgent need to the overall analysis system of the specialty that sets up branches, applying by infosystem, helps to promote branch office's hot line job management level.
Summary of the invention
The present invention is in order to solve above-mentioned existing technical disadvantages, a kind of integrated management analytic system based on mass data is provided, it is by storing the mass data of hot line job professional domain and analyze, for improving hot line job processing quality and decision support provides fast, efficient, the means of aid decision making support reliably.
For achieving the above object, the invention provides a kind of integrated management analytic system based on mass data, comprise the Internet connected successively, fire wall, core switch, load-balanced server, switch and server cluster, described server cluster comprises multiple servers, data scatter is stored in multiple stage independently on server, multiple stage independently server shares storage load, location server is utilized to locate storage information, adopt the computation model based on MapReduce, the computing power that can be made full use of idle computing machine by Distributed Calculation is divided into multiple part the problem that needs huge computing power to solve, and then these parts are distributed to multiple servers process, these result of calculations integrate and obtain net result the most at last.
Further, described system adopts the framework of distributed deployment, adopts load-balancing technique, the load of each station server balanced, thus ensures the access speed that each user asks.
Described load-balancing technique is request user sent, and is forwarded on the little server of load, responds the HTTP request of user, analysis request simultaneously, and be forwarded on web front end machine after systematic analysis.
Described system adopts distributed task dispatching technology, described distributed task dispatching technology adopts distributed task scheduling queue service, is used for performing user task in asynchronous HTTP mode, described system creation task queue, and insert task, the task of inserting queue can perform immediately.
A station server is selected for a post as director server in described server cluster, the increase and decrease of server in the service state of other server and server cluster in this director server management server cluster, once there be machine not provide service, director server redistributes service strategy.
The present invention has following beneficial effect:
1, by infosystem, hot line job specialized standard is carried out standardization solidification and management, realize the robotization of hot line job, intelligent management, to improve standardized level and the efficiency of management of hot line job management.
2, shared and training intercourse platform by the professional knowledge that Information System configuration is perfect, to help hot line job personnel to improve professional skill, reduce operating risk;
3, set up the specialty analysis system analyzed based on mass data, integrated management and analytical applications are carried out to the Various types of data of the magnanimity produced in operation process, to equipment state evaluate and analyze, there is vital role.
4, meet the requirements such as information-based application covering, applying by infosystem, helps to promote branch office's hot line job management level.
Accompanying drawing explanation
In order to be illustrated more clearly in the embodiment of the present invention or technical scheme of the prior art, be briefly described to the accompanying drawing used required in embodiment or description of the prior art below, apparently, accompanying drawing in the following describes is only some embodiments of the present invention, for those of ordinary skill in the art, under the prerequisite not paying creative work, other accompanying drawing can also be obtained according to these accompanying drawings.
Fig. 1 is physical network arrangement block diagram of the present invention;
Fig. 2 is invention distributed task dispatching schematic diagram.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, be clearly and completely described the technical scheme in the embodiment of the present invention, obviously, described embodiment is only the present invention's part embodiment, instead of whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art, not making the every other embodiment obtained under creative work prerequisite, belong to the scope of protection of the invention.
See Fig. 1, Fig. 2, patent of the present invention provides a kind of integrated management analytic system based on mass data, comprise the Internet 1 connected successively, fire wall 2, core switch 3, load-balanced server 4, switch 5 and server cluster 6, in the process of hot line job management, a large amount of structural datas and unstructured data can be accumulated, in traditional management method, the difficult problem depositing and read a management always of data, so we adopt the storage scheme based on distributed file system in this patent, data scatter is stored in multiple stage independently on equipment.Traditional network store system adopts the storage server concentrated to deposit all data, and storage server becomes the bottleneck of system performance, is also the focus of reliability and security, can not meet the needs of Mass storage application.Distributed network storage system adopts extendible system architecture, utilizes multiple stage storage server to share storage load, utilizes location server to locate storage information, and it not only increases the reliability of system, availability and access efficiency, is also easy to expansion.From the angle that the client of file system uses, it is exactly the file system of a standard, provides a series of API, carries out the establishment of file or catalogue, movement, deletion thus, and to operations such as the read-writes of file.Realize from inside, distributed system is then no longer the same with generic file system is in charge of local disk, and its file content and bibliographic structure are all not necessarily stored on local disk, but can by Internet Transmission on far end system.Meanwhile, same file stores just on a machine, but on cluster machine distributed storage, collaboratively provide service.
Mass data analyze, computation model: solve the storage of data in the ladder of management employing distributed storage and search problem, we also need a set of can process these large data analysis, computation model.So we adopt the computation model based on MapReduce, the computing power that can be made full use of idle computing machine by Distributed Calculation is divided into many little parts the problem that needs very huge computing power to solve, then these parts are distributed to many computing machines to process, finally these result of calculations are integrated and obtain final result.
Load-balancing technique: system adopts the framework of distributed deployment, this just needs the load of each station server balanced, thus ensures each access speed of asking.Realize agency and the forwarding of user's request, by the request that user sends, be forwarded to afterwards by analysis on the relatively little server of load.Load balancing is responsible for the HTTP request responding user, analysis request, and is forwarded on the Web server 7 of front end.
Distributed task dispatching technology: adopt distributed task scheduling queue service, be used for performing user task in asynchronous HTTP mode.The queue of online management platform creation task, and insert task, the task of inserting queue can perform immediately.Task queue service is that distributed environment is disposed, and have high reliability, each task work is mutually isolated, and initiatively bears tasks carrying.
Further, a station server is selected for a post as director server in described server cluster, the increase and decrease of server in the service state of other server and server cluster in this director server management server cluster, once there be machine not provide service, director server redistributes service strategy.
The foregoing is only preferred embodiment of the present invention, not in order to limit the present invention, within the spirit and principles in the present invention all, any amendment done, equivalent replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (5)
1. based on the integrated management analytic system of mass data, it is characterized in that: comprise the Internet connected successively, fire wall, core switch, load-balanced server, switch and server cluster, described server cluster comprises multiple servers, data scatter is stored in multiple stage independently on server, multiple stage independently server shares storage load, location server is utilized to locate storage information, adopt the computation model based on MapReduce, the computing power that can be made full use of idle computing machine by Distributed Calculation is divided into multiple part the problem that needs huge computing power to solve, and then these parts are distributed to multiple servers process, these result of calculations integrate and obtain net result the most at last.
2. the integrated management analytic system based on mass data according to claim 1, it is characterized in that: described system adopts the framework of distributed deployment, adopt load-balancing technique, the load of each station server balanced, thus ensure the access speed that each user asks.
3. the integrated management analytic system based on mass data according to claim 2, it is characterized in that: described load-balancing technique is request user sent, be forwarded on the little server of load after systematic analysis, respond the HTTP request of user simultaneously, analysis request, and be forwarded on web front end machine.
4. the integrated management analytic system based on mass data according to claim 3, it is characterized in that: described system adopts distributed task dispatching technology, described distributed task dispatching technology adopts distributed task scheduling queue service, be used for performing user task in asynchronous HTTP mode, described system creation task queue, and insert task, the task of inserting queue can perform immediately.
5. the integrated management analytic system based on mass data according to claim 1, it is characterized in that: select for a post a station server in described server cluster as director server, the increase and decrease of server in the service state of other server and server cluster in this director server management server cluster, once there be machine not provide service, director server redistributes service strategy.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510046328.1A CN104504165A (en) | 2015-01-29 | 2015-01-29 | Mass data based comprehensive management analysis system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510046328.1A CN104504165A (en) | 2015-01-29 | 2015-01-29 | Mass data based comprehensive management analysis system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104504165A true CN104504165A (en) | 2015-04-08 |
Family
ID=52945562
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510046328.1A Pending CN104504165A (en) | 2015-01-29 | 2015-01-29 | Mass data based comprehensive management analysis system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104504165A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106708826A (en) * | 2015-07-30 | 2017-05-24 | 中兴通讯股份有限公司 | Data processing method and apparatus, and data query method and apparatus |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103049482A (en) * | 2012-11-30 | 2013-04-17 | 中国电力科学研究院 | Method for achieving data fusion storage in distributed heterogeneous system |
| CN103560943A (en) * | 2013-10-31 | 2014-02-05 | 北京邮电大学 | Network analytic system and method supporting real-time mass data processing |
| CN103955509A (en) * | 2014-04-30 | 2014-07-30 | 广西电网公司电力科学研究院 | Quick search method for massive electric power metering data |
| CN104283318A (en) * | 2014-10-24 | 2015-01-14 | 国家电网公司 | Power equipment integrated monitoring and early warning system based on big data and analysis method thereof |
-
2015
- 2015-01-29 CN CN201510046328.1A patent/CN104504165A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103049482A (en) * | 2012-11-30 | 2013-04-17 | 中国电力科学研究院 | Method for achieving data fusion storage in distributed heterogeneous system |
| CN103560943A (en) * | 2013-10-31 | 2014-02-05 | 北京邮电大学 | Network analytic system and method supporting real-time mass data processing |
| CN103955509A (en) * | 2014-04-30 | 2014-07-30 | 广西电网公司电力科学研究院 | Quick search method for massive electric power metering data |
| CN104283318A (en) * | 2014-10-24 | 2015-01-14 | 国家电网公司 | Power equipment integrated monitoring and early warning system based on big data and analysis method thereof |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106708826A (en) * | 2015-07-30 | 2017-05-24 | 中兴通讯股份有限公司 | Data processing method and apparatus, and data query method and apparatus |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107864222B (en) | Industrial big data computing system based on PaaS platform | |
| CN105045856B (en) | A kind of big data remote sensing satellite data processing system based on Hadoop | |
| CN105893628A (en) | Real-time data collection system and method | |
| US20130326388A1 (en) | Power grid visualization system and method based on three-dimensional gis technology | |
| CN107612984B (en) | Big data platform based on internet | |
| CN103403683A (en) | Capabilities based routing of virtual data center service request | |
| CN104239144A (en) | Multilevel distributed task processing system | |
| CN107645410A (en) | A kind of virtual machine management system and method based on OpenStack cloud platforms | |
| CN104519140A (en) | Server system for distributed parallel computing and management method thereof | |
| CN108510081A (en) | machine learning method and platform | |
| CN204740299U (en) | Electric energy quality intelligent monitoring system based on cloud calculates | |
| CN104298761B (en) | The implementation method that master data is matched between a kind of heterogeneous software system | |
| Achar | Requirement of Cloud Analytics and Distributed Cloud Computing: An Initial Overview | |
| Manate et al. | Optimizing cloud resources allocation for an Internet of Things architecture | |
| CN104504165A (en) | Mass data based comprehensive management analysis system | |
| Zhang et al. | A novel topology optimization theory and parallel data analysis model based resource scheduling algorithm for cloud computing | |
| Devi et al. | A graph-based mathematical model for an efficient load balancing and fault tolerance in cloud computing | |
| Nandwani et al. | Weight-based data center selection algorithm in cloud computing environment | |
| CN115439015A (en) | Local area power grid data management method, device and equipment based on data middleboxes | |
| CN104391949A (en) | Data dictionary based wide area data resource management method | |
| Jeong et al. | The cloud storage model for manufacturing system in global factory automation | |
| CN110019445B (en) | Data synchronization method and device, computing equipment and storage medium | |
| CN104320457A (en) | Network invoice system based on distributed management | |
| CN203911986U (en) | Cross-platform integrated data processing system | |
| CN104579833B (en) | A kind of FLASH P2P Streaming Media method of real-time based on Hadoop |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20150408 |
|
| RJ01 | Rejection of invention patent application after publication |