CN105630652A - Real-time big data platform Storm oriented runtime three-dimensional visualization system - Google Patents
Real-time big data platform Storm oriented runtime three-dimensional visualization system Download PDFInfo
- Publication number
- CN105630652A CN105630652A CN201610084893.1A CN201610084893A CN105630652A CN 105630652 A CN105630652 A CN 105630652A CN 201610084893 A CN201610084893 A CN 201610084893A CN 105630652 A CN105630652 A CN 105630652A
- Authority
- CN
- China
- Prior art keywords
- storm
- state
- big data
- real
- data platform
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
- G06F11/323—Visualisation of programs or trace data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45504—Abstract machines for programme code execution, e.g. Java virtual machine [JVM], interpreters, emulators
- G06F9/45508—Runtime interpretation or emulation, e g. emulator loops, bytecode interpretation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/805—Real-time
Abstract
The invention provides a real-time big data platform Storm oriented runtime three-dimensional visualization system.The technical scheme includes that the real-time big data platform Storm oriented runtime three-dimensional visualization system periodically interacts with main nodes of Storm cloud environments, states including the numbers and resource consumption of cluster nodes, physical mapping from worker and task threads to Storm cloud, resource consumption of each task and static structures and dynamic properties of topological structures can be sensed, and internal running states of the Storm cloud environments can be monitored; the obtained states can be visualized in real time by the aid of JOGL open source libraries of an OpenGL (open graphics library), and the JOGL open source libraries are implemented on Java virtual machines.The real-time big data platform Storm oriented runtime three-dimensional visualization system has the advantages that the real-time big data platform Storm oriented runtime three-dimensional visualization system is low in resource consumption, and resource competition between the real-time big data platform Storm oriented runtime three-dimensional visualization system and the tasks run by Storm clusters can be prevented.
Description
Technical field
The present invention relates to cloud computing, big data calculate, streaming calculates, data visualization field, are specifically related to three-dimension visible sysem during a kind of operation towards in real time big data platform Storm.
Background technology
The internal real-time visual of Storm cloud environment for perception cloud environment health status is and important.
First, Storm is as magnanimity real-time stream processing platform, and it must assure that high available, high reliability and high usability, excavates Storm cluster internal state, and visualize for guarantee high available and high reliability it is critical that.
Secondly, no matter Storm cloud cluster is that the deployment from task to physical node all has the characteristic of highly distributed from node scale or flow data scale, and this makes the state obtaining whole cloud environment become very difficult. In addition, three-dimensional visualization has the characteristics such as speed is slow, expense big, portable difference, brings lot of challenges to the exploitation of three-dimension visible sysem during the operation towards big data platform Storm in real time.
Summary of the invention
The present invention proposes three-dimension visible sysem during a kind of operation towards in real time big data platform Storm, solves the problem that the state obtaining whole cloud environment in prior art is very difficult.
The technical scheme is that and be achieved in that:
Three-dimension visible sysem during a kind of operation towards in real time big data platform Storm, periodically mutual with Storm cloud environment host node, perception clustered node number and resource consumption, worker and mission thread are to the state of the physical mappings of Storm cloud, each task resource consumption, the static structure of topological structure and dynamic attribute, it is achieved the monitoring to Storm cloud environment internal operation state; Meanwhile, the OpenGL JOGL realized on a java virtual machine storehouse of increasing income is utilized to realize the real-time visual to the state obtained.
Alternatively, the Nimbus host node run on Storm cloud environment provides the api interface calling self for the external world, first pass through designated host IP and Nimbus process place port open remote procedure call client, the state of client-access physical cluster and the state of service topology.
Alternatively, computing node in cluster periodically sends oneself the state Nimbus process to host node, the state of its in store each computing node and the topological state operated on computing node, the client being connected to Nimbus process periodically calls the Nimbus api interface provided to obtain the state of its preservation, thus realizing the state aware of cluster.
The invention has the beneficial effects as follows:
(1) there is the feature that resource consumption is few, resource will not be fought for the task that Storm cluster runs.
(2) this system is run on a java virtual machine, but the three dimensional display of its bottom is based on OpenGL exploitation, by JNI technological sourcing dynamic link library, it is achieved that acceptable response speed.
Accompanying drawing explanation
In order to be illustrated more clearly that the embodiment of the present invention or technical scheme of the prior art, the accompanying drawing used required in embodiment or description of the prior art will be briefly described below, apparently, accompanying drawing in the following describes is only some embodiments of the present invention, for those of ordinary skill in the art, under the premise not paying creative work, it is also possible to obtain other accompanying drawing according to these accompanying drawings.
The topoisomerase ��� of three-dimension visible sysem when Fig. 1 is a kind of operation towards in real time big data platform Storm of the present invention;
Three-dimension visible sysem schematic diagram when Fig. 2 is a kind of operation towards in real time big data platform Storm of the present invention.
Detailed description of the invention
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is clearly and completely described, it is clear that described embodiment is only a part of embodiment of the present invention, rather than whole embodiments. Based on the embodiment in the present invention, the every other embodiment that those of ordinary skill in the art obtain under not making creative work premise, broadly fall into the scope of protection of the invention.
The present invention proposes three-dimension visible sysem during a kind of operation towards in real time big data platform Storm, first, opens Storm real-time cloud environment, and submits to topology to cluster; It follows that open Storm deployable state catcher; Then, three-dimensional visualization server is opened; Subsequently, it is judged that and update Storm cluster state, it is judged that whether Storm deployable state is latest data; Finally, the run time behaviour of the physical characteristic of Dynamic Announce cluster, deployable state and task in three dimensions.
Below in conjunction with Figure of description 1 and accompanying drawing 2 to the present invention towards the operation of big data platform Storm in real time time three-dimension visible sysem be described in detail.
As it is shown in figure 1, under a cloud real time computing environment, each task runs respective service computation logic, the dependency existed between task represents mainly through logical relation and two aspects of physical relation. Logical relation aspect refers to the data communication between calculating task, as it is shown in figure 1, the data flow between the oriented arrow expression task between task, this flow direction also embodies line between ball in fig. 2 simultaneously. Physical relation aspect refers to the physical location of task and task place, and the position relationship between task specifically includes that whether be in same cloud, if be in Same Physical machine, if be in same virtual machine. Dotted line in Fig. 1 represents calculating task, and (virtual machine moves in the one or more ports in physical node with physical node COM1, inside Storm real-time cloud computing environment, what represent is a worker process) between mapping relations, this relation line between ball and cube in fig. 2 also shows. Ball represents the calculating task of Storm cloud in fig. 2, and cube represents a certain virtual machine (inside Storm real-time cloud computing environment, expression is a worker process) in physical node; The logical relation between line expression task between ball, the line between ball and cube represents the physical mappings between task and virtual machine, and the line between cube represents that its virtual machine linked is in Same Physical node.
As in figure 2 it is shown, the present invention propose a kind of operation towards in real time big data platform Storm time three-dimension visible sysem, its operation principle is in that:
It is by periodically mutual with Storm cloud environment host node, perception clustered node number and resource consumption, worker and mission thread to the physical mappings of Storm cloud, each task resource consumption, topological structure static structure and dynamic attribute etc. in interior state, it is achieved the monitoring to Storm cloud environment internal operation state; Meanwhile, the JOGL that OpenGL three-dimensional visualization technique realizes on a java virtual machine storehouse of increasing income is utilized to realize the real-time visual to above-mentioned obtained state.
The Nimbus host node run on Storm cloud environment provides the api interface calling self for the external world, the process that the system of the present invention is mutual with Storm cloud environment host node, particularly as follows: first pass through designated host IP and Nimbus process place port open remote procedure call client, such client can access the state of physical cluster and the state of service topology.
Computing node in cluster can periodically send oneself the state Nimbus process to host node, the state of its in store each computing node and the topological state operated on computing node, the client being connected to Nimbus process can periodically call the Nimbus api interface provided to obtain the state of its preservation, thus realizing the state aware of cluster.
Along with the submission of topology and end, the additions and deletions of physical node, the scheduling that flows into the expansion of Storm cloud environment data scale and minimizing, task and worker migrate, the present invention towards the operation of big data platform Storm in real time time three-dimension visible sysem can correctly these changes be shown in above Visualization Platform.
The present invention towards the operation of in real time big data platform Storm time three-dimension visible sysem have the advantages that resource consumption is few, resource will not be fought for the task that Storm cluster runs; This system is run on a java virtual machine, but the three dimensional display of its bottom is based on OpenGL exploitation, by JNI technological sourcing dynamic link library, it is achieved that acceptable response speed.
The foregoing is only presently preferred embodiments of the present invention, not in order to limit the present invention, all within the spirit and principles in the present invention, any amendment of making, equivalent replacement, improvement etc., should be included within protection scope of the present invention.
Claims (3)
1. three-dimension visible sysem during an operation towards in real time big data platform Storm, it is characterized in that, periodically mutual with Storm cloud environment host node, perception clustered node number and resource consumption, worker and mission thread are to the state of the physical mappings of Storm cloud, each task resource consumption, the static structure of topological structure and dynamic attribute, it is achieved the monitoring to Storm cloud environment internal operation state; Meanwhile, the OpenGL JOGL realized on a java virtual machine storehouse of increasing income is utilized to realize the real-time visual to the state obtained.
2. three-dimension visible sysem during operation towards big data platform Storm in real time as claimed in claim 1, it is characterized in that, the Nimbus host node run on Storm cloud environment provides the api interface calling self for the external world, first pass through designated host IP and Nimbus process place port open remote procedure call client, the state of client-access physical cluster and the state of service topology.
3. three-dimension visible sysem during operation towards big data platform Storm in real time as claimed in claim 1, it is characterized in that, computing node in cluster periodically sends oneself the state Nimbus process to host node, the state of its in store each computing node and the topological state operated on computing node, the client being connected to Nimbus process periodically calls the Nimbus api interface provided to obtain the state of its preservation, thus realizing the state aware of cluster.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610084893.1A CN105630652A (en) | 2016-02-02 | 2016-02-02 | Real-time big data platform Storm oriented runtime three-dimensional visualization system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610084893.1A CN105630652A (en) | 2016-02-02 | 2016-02-02 | Real-time big data platform Storm oriented runtime three-dimensional visualization system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105630652A true CN105630652A (en) | 2016-06-01 |
Family
ID=56045625
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610084893.1A Pending CN105630652A (en) | 2016-02-02 | 2016-02-02 | Real-time big data platform Storm oriented runtime three-dimensional visualization system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105630652A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106534251A (en) * | 2016-09-23 | 2017-03-22 | 郑州云海信息技术有限公司 | Task visual uploading and starting method based on Storm |
| CN106648928A (en) * | 2016-11-29 | 2017-05-10 | 成都广达新网科技股份有限公司 | Method and device for inter-process communication |
| CN111382022A (en) * | 2018-12-27 | 2020-07-07 | 北京神州泰岳软件股份有限公司 | Method and device for monitoring real-time streaming computing platform, electronic equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103701906A (en) * | 2013-12-27 | 2014-04-02 | 北京奇虎科技有限公司 | Distributed real-time calculation system and data processing method thereof |
| CN103763378A (en) * | 2014-01-24 | 2014-04-30 | 中国联合网络通信集团有限公司 | Task processing method and system and nodes based on distributive type calculation system |
| CN104615526A (en) * | 2014-12-05 | 2015-05-13 | 北京航空航天大学 | Monitoring system of large data platform |
| CN104639466A (en) * | 2015-03-05 | 2015-05-20 | 北京航空航天大学 | Dynamic priority safeguard method for application network bandwidth based on Storm real-time flow computing framework |
-
2016
- 2016-02-02 CN CN201610084893.1A patent/CN105630652A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103701906A (en) * | 2013-12-27 | 2014-04-02 | 北京奇虎科技有限公司 | Distributed real-time calculation system and data processing method thereof |
| CN103763378A (en) * | 2014-01-24 | 2014-04-30 | 中国联合网络通信集团有限公司 | Task processing method and system and nodes based on distributive type calculation system |
| CN104615526A (en) * | 2014-12-05 | 2015-05-13 | 北京航空航天大学 | Monitoring system of large data platform |
| CN104639466A (en) * | 2015-03-05 | 2015-05-20 | 北京航空航天大学 | Dynamic priority safeguard method for application network bandwidth based on Storm real-time flow computing framework |
Non-Patent Citations (3)
| Title |
|---|
| ARRAY020: "《http://blog.csdn.net/damacheng/article/details/42706605》", 14 January 2015, CSDN博客 * |
| 陈星雨: "基于OPENGL和GDAL的卫星遥感图像处理系统的研究", 《中国优秀硕士论文全文数据库 信息科技辑》 * |
| 马延辉等: "《Storm企业级应用 实战、运维和调优》", 30 June 2015, 机械工作出版社 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106534251A (en) * | 2016-09-23 | 2017-03-22 | 郑州云海信息技术有限公司 | Task visual uploading and starting method based on Storm |
| CN106648928A (en) * | 2016-11-29 | 2017-05-10 | 成都广达新网科技股份有限公司 | Method and device for inter-process communication |
| CN111382022A (en) * | 2018-12-27 | 2020-07-07 | 北京神州泰岳软件股份有限公司 | Method and device for monitoring real-time streaming computing platform, electronic equipment and storage medium |
| CN111382022B (en) * | 2018-12-27 | 2024-02-20 | 北京神州泰岳软件股份有限公司 | Method, device, electronic equipment and storage medium for monitoring real-time stream computing platform |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8413166B2 (en) | Multithreaded physics engine with impulse propagation | |
| US20190164518A1 (en) | Dynamic jitter and latency-tolerant rendering | |
| US8773449B2 (en) | Rendering of stereoscopic images with multithreaded rendering software pipeline | |
| US10063427B1 (en) | Visualizing and interacting with resources of an infrastructure provisioned in a network | |
| US8627329B2 (en) | Multithreaded physics engine with predictive load balancing | |
| US10300382B1 (en) | Three dimensional terrain modeling | |
| US11456941B2 (en) | Extensible network traffic engineering platform for increasing network resiliency in cloud applications | |
| CN103914868B (en) | Method for mass model data dynamic scheduling and real-time asynchronous loading under virtual reality | |
| US9270523B2 (en) | Reconfiguring interrelationships between components of virtual computing networks | |
| WO2018037200A1 (en) | Simulation systems and methods | |
| CN105630652A (en) | Real-time big data platform Storm oriented runtime three-dimensional visualization system | |
| US11550621B2 (en) | Distributable and customizable load-balancing of data-associated computation via partitions and virtual processes | |
| CN110489204A (en) | A kind of big data platform architecture system based on container cluster | |
| US8564600B2 (en) | Streaming physics collision detection in multithreaded rendering software pipeline | |
| US20190095560A1 (en) | Commands For Simulation Systems And Methods | |
| CN113572815A (en) | Communication technology method, system and medium for crossing heterogeneous platforms | |
| US10552513B1 (en) | Computer system entity rendering system | |
| US11533367B2 (en) | Simulation systems and methods using query-based interest | |
| CN113297262A (en) | Data flow graph layout method and device | |
| CN114510321A (en) | Resource scheduling method, related device and medium | |
| CN106886477A (en) | Threshold setting method and device are monitored in a kind of cloud system | |
| US11552859B2 (en) | Interactive graphical model-based configuration control of networked physical assets using logical transactional configuration change rules | |
| Zheng et al. | Virtual technology of cache and real-time big data distribution in cloud computing big data center | |
| KR101576107B1 (en) | Systems and methods for data synchronization in a network application | |
| CN106899475A (en) | A kind of method of the method for integrating tunnel resource, device and treatment message |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160601 |