CN105824618A - Real-time message processing method for Storm - Google Patents
Real-time message processing method for Storm Download PDFInfo
- Publication number
- CN105824618A CN105824618A CN201610135228.0A CN201610135228A CN105824618A CN 105824618 A CN105824618 A CN 105824618A CN 201610135228 A CN201610135228 A CN 201610135228A CN 105824618 A CN105824618 A CN 105824618A
- Authority
- CN
- China
- Prior art keywords
- topology
- storm
- node
- nimbus
- place
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 13
- 238000000034 method Methods 0.000 claims abstract description 79
- 229940126655 NDI-034858 Drugs 0.000 claims abstract description 40
- 241000290929 Nimbus Species 0.000 claims abstract description 40
- 238000012545 processing Methods 0.000 claims abstract description 28
- 230000006870 function Effects 0.000 description 4
- 238000012423 maintenance Methods 0.000 description 2
- 206010008190 Cerebrovascular accident Diseases 0.000 description 1
- 208000006011 Stroke Diseases 0.000 description 1
- 238000009825 accumulation Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000008034 disappearance Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 238000012856 packing Methods 0.000 description 1
- 238000011144 upstream manufacturing Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a real-time message processing method used for Storm, belonging to the technical field of Storm and aiming at solving the problems that data is not lost in the process of message processing and the whole processing system has good expansibility. The technical scheme is as follows: all Topology submissions are carried out on client nodes of Storm, and control nodes of Nimbus are distributed to working nodes of other Supervisors for processing. Nimbus firstly segments the submitted Topology, divides the Topology into individual tasks, submits the information related to the tasks and the Supervisor to the zookeeper cluster, and the Supervisor asks the own tasks on the zookeeper cluster and informs the own Worker process to process the tasks.
Description
Technical field
The present invention relates to a kind of Storm technical field, a kind of real-time message processing method used about Storm.
Background technology
Along with the high speed development of internet, applications, the data volume of enterprise's accumulation is increasing, more and more.Along with the appearance of the correlation techniques such as GoogleMapReduce, Hadoop, processing large-scale data and become simply, but these data processing techniques are not the most real-time systems, their design object is not to calculate in real time.The most real-time calculating system and system based on batch processing model (such as Hadoop) are essentially different.
But being as the quick growth of big data service, the calculating in real time processed for large-scale data becomes a kind of operational demand, lacks " real-time Hadoop system " and has become as a huge disappearance in whole big data ecosystem.Storm occurs just under such requirement background, and Storm meets this demand well.Storm is cluster assembly, and the cluster of Storm is formally seen closely similar with the cluster of Hadoop, is also to use client/server.But that operate above at Hadoop is the Job of MapReduce, and that operate above at Storm is Topology.They are very different.Key difference between them is, MapReduceJob has startup, runs to the process that eventually terminates, and a Topology is upon actuation, can forever run.Two kinds of nodes are had: control node and working node inside the cluster of Storm.Controlling node and operate above a background process Nimbus, its effect is similar to the JobTracker inside Hadoop.Nimbus is responsible for distribution inside cluster and performs code, shares out the work to working node, and the execution state of monitor task.Each working node operates above a finger daemon being called Supervisor.Supervisor can monitor the work distributing to oneself place machine, the on/off progress of work as required.Each progress of work performs a subset of a Topology, and a Topology run is made up of a lot of progresses of work operated on a lot of machine.
Before Storm occurs, for needing the task of realizing calculating, developer needs the real time processing network that one message queue of manual maintenance and Message Processing person are formed, and Message Processing person takes out message from message queue and processes, the most more new database, sends messages to other queues.All these operations are required for developer oneself and realize.There is following defect in the pattern of this programming realization:
1, dull property: developer needs to spend the most of the time to go to configure how message sends, and where message is sent to, and how to dispose the processor of message, how to dispose the intermediate processing nodes etc. of message.If using Storm process, then developer has only to little message processing logic code, such developer just can be absorbed in the exploitation of service logic, thus substantially increases the efficiency of exploitation real time computation system；
2、Vulnerability: program is the most healthy and the strongest, developer needs the correct operation oneself writing code to ensure all of Message Processing person and message queue；
3, scalability is poor: when the treatable message of Message Processing person reaches oneself treatable peak value, it is necessary to shunt message stream, at this moment need to configure new Message Processing person, to allow them process shunting message.
For the real-time system that needs process a large amount of message streams, Message Processing is the basis calculated in real time all the time, and the last of Message Processing is exactly to the combination between message queue and Message Processing person.The core of Message Processing is how not lose data during Message Processing, and whole processing system can be made to have good autgmentability, so as to process bigger message stream.
Summary of the invention
The technical assignment of the present invention is for above weak point, it is provided that a kind of real-time message processing method used about Storm, solves how not lose during Message Processing data, and can make the problem that whole processing system has good autgmentability.
The technical solution adopted for the present invention to solve the technical problems is:
A kind of real-time message processing method used about Storm, in the server cluster using Storm, uses Storm and relies on external resource Zookeeper cluster and carry out real-time message processing；Storm has Nimbus and Supervisor, the control node of server cluster is disposed Nimbus, each working node of server cluster all disposes Supervisor；Storm submits to the program run to be that Topology, Topology are made up of Spout and Bolt；Comprise the steps:
(1), the submission of Topology is carried out on control node, first code is stored on the control node at Nimbus place, afterwards, the configuration that current Storm is run generates file and is put in the control node at Nimbus place, also has the Topology code file after serializing in the control node at Nimbus place simultaneously；
(2), Nimbus when setting Spouts and Bolts associated by Topology, the number of the worker process of current Spout and Bolt is set simultaneously, executor number and task number is i.e. set；According to the number of worker process, average runs worker course allocation to each working node；Worker process is run on the working node at which supervisor place and is determined by storm itself；
(3), after task distributes, the heartbeat message of worker process is submitted to zookeeper cluster by the node that controls at Nimbus place；
(4), the working node at Supervisor place continuous poll zookeeper cluster, the heartbeat message (incidence relation between the task allocation information of all Topology, code storage catalogue and task) of worker process is saved in zookeeper, the working node at Supervisor place is by the content in poll zookeeper, get the task of oneself, start worker process and run；
(5), after Topology runs, constantly sending Stream by Spouts and flow, constantly processed the Stream stream received by Bolts, Stream stream is unbounded.
In step (1), the submission of Topology is carried out on control node, first code is stored under the inbox catalogue controlling node at Nimbus place, afterwards, the configuration that current Storm is run generates file stormconf.ser and is put in the stormdist catalogue controlling node at Nimbus place, also has the Topology code file after serializing in stormdist catalogue simultaneously；
In step (2), the summation of the task number of a Topology is consistent with the summation of executor number.
In step (3), being provided with workerbeats node in zookeeper cluster, workerbeats node stores the heartbeat message of all worker processes of current Topology；
In step (4), the working node at Supervisor place can continuous poll zookeeper cluster, save the task allocation information of all Topology in the assignments node of zookeeper, code stores the incidence relation between catalogue and task, the working node at Supervisor place is by the content of the assignments node of poll zookeeper, get the task of oneself, start worker process and run.
Zookeeper be one for Distributed Application provide Consistency service software, it is provided that function include: configuring maintenance, domain name service, distributed synchronization, group service.
Storm has Nimbus and Supervisor, the control node of server cluster is disposed Nimbus, each working node of server cluster all disposes Supervisor；Nimbus is responsible for inside server cluster sending code, and distribution Worker process is to working node, and monitors working node state；Supervisor monitors the work of place working node, starts as required or closes Worker process；The Worker process of Nimbus and Supervisor and operation is all saved in heartbeat message on Zookeeper cluster；Nimbus, according to the heartbeat message on Zookeerper cluster and task run situation, is scheduling and distributes Worker process.
The message unit that Storm submits to the program run to be the minimum that Topology, Topology process is a Tuple, the namely array of an any object；Topology is made up of Spout and Bolt；Spout is the node sending Tuple；Bolt can arbitrarily subscribe to the Tuple that certain Spout or Bolt sends；Spout and Bolt is referred to as component.
A kind of real-time message processing method used about Storm of the present invention is compared to the prior art, have the advantages that the real-time calculating that Storm can write in a computer cluster easily and extend complexity, process in real time, it is ensured that each message can be processed；And it is quickly, in a little cluster, per second can process millions of message；And any programming language can be used to develop.
Accompanying drawing explanation
The present invention is further described below in conjunction with the accompanying drawings.
Accompanying drawing 1 is the Storm Organization Chart of a kind of real-time message processing method used about Storm.
Detailed description of the invention
Below in conjunction with the accompanying drawings and specific embodiment the invention will be further described.
A kind of real-time message processing method used about Storm of the present invention, in the server cluster using Storm, uses Storm and relies on external resource Zookeeper cluster and carry out real-time message processing；Storm has Nimbus and Supervisor, the control node of server cluster is disposed Nimbus, each working node of server cluster all disposes Supervisor；Storm submits to the program run to be that Topology, Topology are made up of Spout and Bolt；Comprise the steps:
(1), the submission of Topology is carried out on control node, first code is stored under the inbox catalogue controlling node at Nimbus place, afterwards, the configuration that current Storm is run generates file stormconf.ser and is put in the stormdist catalogue controlling node at Nimbus place, also has the Topology code file after serializing in stormdist catalogue simultaneously；
(2), Nimbus when setting Spouts and Bolts associated by Topology, the number of the worker process of current Spout and Bolt is set simultaneously, executor number and task number is i.e. set；According to the number of worker process, average runs worker course allocation to each working node；Worker process is run on the working node at which supervisor place and is determined by storm itself；The summation of the task number of one Topology is consistent with the summation of executor number
(3), after task distributes, the heartbeat message of worker process is submitted to zookeeper cluster by the node that controls at Nimbus place；Being provided with workerbeats node in zookeeper cluster, workerbeats node stores the heartbeat message of all worker processes of current Topology；
(4), the working node at Supervisor place can continuous poll zookeeper cluster, the heartbeat message (incidence relation between the task allocation information of all Topology, code storage catalogue and task) of worker process is saved in the assignments node of zookeeper, the working node at Supervisor place is by the content of the assignments node of poll zookeeper, get the task of oneself, start worker process and run
(5), after Topology runs, constantly sending Stream by Spouts and flow, constantly processed the Stream stream received by Bolts, Stream stream is unbounded.
Final step can continual execution, unless manually terminated Topology.
The submission of all Topology tasks must carry out (needing to configure storm.yaml file) on Storm client node, the control node at Nimbus place the working node distributing to other Supervisor places processes.First the Topology of submission is carried out burst by Nimbus, it is divided into Task one by one, and information relevant for Task with Supervisor is submitted on zookeeper cluster, Supervisor removes to claim on zookeeper cluster the Task of oneself, notifies that the Worker process of oneself carries out the process of Task.
Below the number order code in Storm is illustrated:
One,Spout
Spout is that the message of Stream produces source, and the realization of Spout assembly can complete by inheriting BaseRichSpout class or other Spout classes, it is also possible to realizes by realizing IRichSpout interface.
publicinterfaceISpoutextendsSerializable{
voidopen(Mapconf,TopologyContextcontext,SpoutOutputCollectorcollector);
voidclose();
voidnextTuple();
voidack(ObjectmsgId);
voidfail(ObjectmsgId);
}。
Open () method, is initial method.
Close (): call when this spout will close.But does not guarantee that it is necessarily called, because supervisor node in the cluster, it is possible to use kill-9 kills worker process.Being only to run in local mode as Storm, ceasing and desisting order if sending, it is ensured that the execution of close.
Ack (ObjectmsgId): the method for readjustment when being successfully processed tuple, it is generally the case that the realization of the method is the message in message queue to be removed, and prevents message-replay.
Fail (ObjectmsgId): the method for readjustment when processing tuple failure, it is generally the case that the realization of the method is to put back to message in message queue then to reset in the time after a while.
NextTuple (): this is the most important method of Spout apoplexy due to endogenous wind.Launch Tuple a to Topology all to be realized by this method.When calling the method, storm sends request to spout, allows spout send tuple (tuple) to follower (ouputcollector).This method should be non-obstruction, so spout sends without tuple, this method should return.NextTuple, ack and fail all in the same thread of spout task by recursive call.When there is no the transmitting of tuple, it should allow nextTuple sleep a shortest time (such as a millisecond), in order to avoid wasting too many CPU.
After inheriting BaseRichSpout, close, activate, deactivate, ack, fail and getComponentConfiguration method need not be realized, be only concerned the part of most basic core.
Under normal circumstances (except Shell and affairs type), it is achieved a Spout, interface IRichSpout can being directly realized by, if being not desired to write unnecessary code, can directly inherit BaseRichSpout.
Two, Bolt
Bolt class receives the Tuple sent by Spout or other upstreams Bolt class, processes it.The realization of Bolt assembly can complete by inheriting BasicRichBolt class or IRichBolt interface etc..
Prepare method: the method is similar with the open method in Spout, the task in a worker calls when initializing in the cluster.It provide the environment that bolt performs.
DeclareOutputFields method: for stating the field (field) comprised in the Tuple that current Bolt sends, similar with in Spout.
Cleanup method: with the close method of ISpout, call before being turned off.The most do not ensure that it necessarily performs.
Execute method: this is a method of most critical in Bolt, can be put in the method for the process of Tuple and carry out.Concrete transmission is completed by emit method.Execute accepts a tuple process, and the ack method (representing successfully) or fail(with the incoming OutputCollector of prepare method represents failure) carry out feedback processing result.
Storm provides IBasicBolt interface, and its purpose is exactly that the Bolt realizing this interface not be used in code offer feedback result, and Storm is internal can automatic feedback success.If you want feeding back unsuccessful really, can dish out FailedException.
Under normal circumstances, realize a Bolt, IRichBolt interface can be realized or inherit BaseRichBolt, if being not desired to oneself result feedback, can realize IBasicBolt interface or inherit BaseBasicBolt, it is effectively equivalent to be automatically obtained collector.emit.ack (inputTuple).
Three, the Topology method of operation
Before starting establishment project, the operator scheme (operationmodes) understanding Storm is critically important.Storm has two kinds of methods of operation, the way of submission of local runtime and distributed way of submission.
The way of submission of local runtime, example:
LocalClustercluster=newLocalCluster();
cluster.submitTopology(TOPOLOGY_NAME,
conf,builder.createTopology());
Thread.sleep(2000);
cluster.shutdown();
Distributed way of submission, example:
StormSubmitter.submitTopology(TOPOLOGY_NAME, conf, builder.createTopology ()).
It should be noted that, after Storm written in code completes, need to be packaged into jar bag and be put in Nimbus operation, the when of packing, the jar relied on is need not all to beat, if else if the storm.jar bag relied on being thrown into, the configuration file mistake that there will be repetition during operation causes Topology to run.Because before Topology runs, the storm.yaml configuration file of this locality can be loaded.
The order run is as follows: stormjarStormTopology.jarmainclass [args].
The order of storm finger daemon:
Nimbus:stormnimbus starts nimbus finger daemon；
Supervisor:stormsupervisor starts supervisor and guards journey；
UI:stormui this will start stormUI finger daemon, for monitoring storm cluster provide a user interface based on web.
DRPC:stormdrpc starts the finger daemon of DRPC.
Storm administration order:
JAR:stormjartopology_jartopology_class [arguments...].
Jar order is for submitting a cluster topology to. it runs the main () method in the topology_class specifying parameter, uploads topology_jar to nimbus, nimbus is published in cluster.Once submit to, storm will activate topology and starts the main () method processing in topology_class, main () method is responsible for calling StormSubmitter.submitTopology () method, and provides the name of a unique topology (cluster).If in a topology Already in cluster having this title, jar order will be failed.Common way is to use command line parameter to specify topology title, in order to topology is named submitting to when.
KILL:stormkilltopology_name [-wwait_time]；
Kill a topology, it is possible to use kill order.It can destroy a topology in a secure manner, first disables topology, allows topology to complete current data stream within the time period waiting topology message.Performing to pass through-w [wait number of seconds] during kill order specifies topology to disable the later waiting time.Same function can also be realized on StormUI interface.
Deactivate:stormdeactivatetopology_name；
When disabling topology, all tuples distributed all can be processed, and the nextTuple method of spouts will not be called.Same function can also be realized on StormUI interface.
Activate:stormactivatetopology_name；
Start a topology disabled.Same function can also be realized on StormUI interface.
Rebalance:stormrebalancetopology_name [-wwait_time] [-nworker_count] [-ecomponent_name=executer_count] ...；
Rebalance makes you redistribute cluster task.This is a most powerful order.Such as, you add node to an operating cluster.Rebalance order will disable topology, then reassigns worker after corresponding time-out time, and restarts topology；
Example: stormrebalancewordcount-topology-w15-n5-esentence-spout=4-esplit-bolt=8.
Also have other administration orders, such as: the newly-built storm project points for attention such as Remoteconfvalue, REPL, Classpath.
In order to develop storm project, inside classpath, need the jar bag of storm.The mode recommended most is to use Maven, manually all of jar bag inside storm release can be added to classpath if not using maven.
By detailed description of the invention above, described those skilled in the art can readily realize the present invention.It is understood that the present invention is not limited to above-mentioned detailed description of the invention.On the basis of disclosed embodiment, described those skilled in the art can the different technical characteristic of combination in any, thus realize different technical schemes.
In addition to the technical characteristic described in description, it is the known technology of those skilled in the art.
Claims (4)
1. the real-time message processing method used about Storm, it is characterised in that in the server cluster using Storm, uses Storm and relies on external resource Zookeeper cluster and carry out real-time message processing；Storm has Nimbus and Supervisor, the control node of server cluster is disposed Nimbus, each working node of server cluster all disposes Supervisor；Storm submits to the program run to be that Topology, Topology are made up of Spout and Bolt；Comprise the steps:
(1), the submission of Topology is carried out on control node, first code is stored on the control node at Nimbus place, afterwards, the configuration that current Storm is run generates file and is put in the control node at Nimbus place, also has the Topology code file after serializing in the control node at Nimbus place simultaneously；
(2), Nimbus when setting Spouts and Bolts associated by Topology, the number of the worker process of current Spout and Bolt is set simultaneously, executor number and task number is i.e. set；According to the number of worker process, worker course allocation is run to each working node；
(3), after task distributes, the heartbeat message of worker process is submitted to zookeeper cluster by the node that controls at Nimbus place；
(4), the working node at Supervisor place continuous poll zookeeper cluster, the heartbeat message of worker process is saved in zookeeper, the working node at Supervisor place is by the content in poll zookeeper, get the task of oneself, start worker process and run；
(5), after Topology runs, constantly sending Stream by Spouts and flow, constantly processed the Stream stream received by Bolts, Stream stream is unbounded.
A kind of real-time message processing method used about Storm the most according to claim 1, it is characterized in that in step (1), the submission of Topology is carried out on control node, first code is stored under the inbox catalogue controlling node at Nimbus place, afterwards, the configuration that current Storm is run generates file stormconf.ser and is put in the stormdist catalogue controlling node at Nimbus place, also has the Topology code file after serializing in stormdist catalogue simultaneously.
A kind of real-time message processing method used about Storm the most according to claim 1, it is characterised in that in step (2), the summation of the task number of a Topology is consistent with the summation of executor number.
A kind of real-time message processing method used about Storm the most according to claim 1, it is characterized in that in step (3), being provided with workerbeats node in zookeeper cluster, workerbeats node stores the heartbeat message of all worker processes of current Topology；
In step (4), the working node at Supervisor place can continuous poll zookeeper cluster, save the task allocation information of all Topology in the assignments node of zookeeper, code stores the incidence relation between catalogue and task, the working node at Supervisor place is by the content of the assignments node of poll zookeeper, get the task of oneself, start worker process and run.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610135228.0A CN105824618A (en) | 2016-03-10 | 2016-03-10 | Real-time message processing method for Storm |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610135228.0A CN105824618A (en) | 2016-03-10 | 2016-03-10 | Real-time message processing method for Storm |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105824618A true CN105824618A (en) | 2016-08-03 |
Family
ID=56987101
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610135228.0A Pending CN105824618A (en) | 2016-03-10 | 2016-03-10 | Real-time message processing method for Storm |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105824618A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106445790A (en) * | 2016-10-12 | 2017-02-22 | 北京集奥聚合科技有限公司 | Counting and account-checking method and device used in distributed real-time computing system |
| CN106843930A (en) * | 2016-12-23 | 2017-06-13 | 江苏途致信息科技有限公司 | Streaming dynamic configuration more new architecture and method based on zookeeper |
| CN107885881A (en) * | 2017-11-29 | 2018-04-06 | 顺丰科技有限公司 | Business datum real-time report, acquisition methods, device, equipment and its storage medium |
| KR20180072295A (en) * | 2016-12-21 | 2018-06-29 | 세림티에스지(주) | Dynamic job scheduling system and method for supporting real-time stream data processing in distributed in-memory environment |
| CN108363619A (en) * | 2018-03-07 | 2018-08-03 | 深圳市酷开网络科技有限公司 | Service procedure control method, server and computer readable storage medium |
| CN110971687A (en) * | 2019-11-29 | 2020-04-07 | 浙江邦盛科技有限公司 | Rail transit flow data processing method |
| TWI690849B (en) * | 2016-10-28 | 2020-04-11 | 香港商阿里巴巴集團服務有限公司 | Method and device for upgrading and closing applications |
| CN111522637A (en) * | 2020-04-14 | 2020-08-11 | 重庆邮电大学 | Storm task scheduling method based on cost benefit |
| CN113688115A (en) * | 2021-08-29 | 2021-11-23 | 中盾创新档案管理(北京)有限公司 | File big data distributed storage system based on Hadoop |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103309903A (en) * | 2012-03-16 | 2013-09-18 | 刘龙 | Position search system and method based on cloud computing |
| CN103716182A (en) * | 2013-12-12 | 2014-04-09 | 中国科学院信息工程研究所 | Failure detection and fault tolerance method and failure detection and fault tolerance system for real-time cloud platform |
| CN104036025A (en) * | 2014-06-27 | 2014-09-10 | 蓝盾信息安全技术有限公司 | Distribution-base mass log collection system |
| CN104268260A (en) * | 2014-10-10 | 2015-01-07 | 中国科学院重庆绿色智能技术研究院 | Method, device and system for classifying streaming data |
| WO2015085961A1 (en) * | 2013-12-09 | 2015-06-18 | 腾讯科技（深圳）有限公司 | User profile configuring method and device |
| CN104916127A (en) * | 2014-03-13 | 2015-09-16 | 深圳市赛格导航科技股份有限公司 | Internet of vehicles distributed real-time traffic condition analysis method and system |
-
2016
- 2016-03-10 CN CN201610135228.0A patent/CN105824618A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103309903A (en) * | 2012-03-16 | 2013-09-18 | 刘龙 | Position search system and method based on cloud computing |
| WO2015085961A1 (en) * | 2013-12-09 | 2015-06-18 | 腾讯科技（深圳）有限公司 | User profile configuring method and device |
| CN103716182A (en) * | 2013-12-12 | 2014-04-09 | 中国科学院信息工程研究所 | Failure detection and fault tolerance method and failure detection and fault tolerance system for real-time cloud platform |
| CN104916127A (en) * | 2014-03-13 | 2015-09-16 | 深圳市赛格导航科技股份有限公司 | Internet of vehicles distributed real-time traffic condition analysis method and system |
| CN104036025A (en) * | 2014-06-27 | 2014-09-10 | 蓝盾信息安全技术有限公司 | Distribution-base mass log collection system |
| CN104268260A (en) * | 2014-10-10 | 2015-01-07 | 中国科学院重庆绿色智能技术研究院 | Method, device and system for classifying streaming data |
Non-Patent Citations (1)
| Title |
|---|
| Z702143700: "《基于Storm的实时大数据处理》", 《百度文库：HTTPS://WENKU.BAIDU.COM/VIEW/EADFDCA8240C844768EAEE33?FROM=SEARCH》 * |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106445790A (en) * | 2016-10-12 | 2017-02-22 | 北京集奥聚合科技有限公司 | Counting and account-checking method and device used in distributed real-time computing system |
| US10678532B2 (en) | 2016-10-28 | 2020-06-09 | Alibaba Group Holding Limited | Method and apparatus for upgrading application |
| TWI690849B (en) * | 2016-10-28 | 2020-04-11 | 香港商阿里巴巴集團服務有限公司 | Method and device for upgrading and closing applications |
| KR20180072295A (en) * | 2016-12-21 | 2018-06-29 | 세림티에스지(주) | Dynamic job scheduling system and method for supporting real-time stream data processing in distributed in-memory environment |
| KR101886072B1 (en) * | 2016-12-21 | 2018-08-08 | 세림티에스지(주) | Dynamic job scheduling system and method for supporting real-time stream data processing in distributed in-memory environment |
| CN106843930A (en) * | 2016-12-23 | 2017-06-13 | 江苏途致信息科技有限公司 | Streaming dynamic configuration more new architecture and method based on zookeeper |
| CN107885881A (en) * | 2017-11-29 | 2018-04-06 | 顺丰科技有限公司 | Business datum real-time report, acquisition methods, device, equipment and its storage medium |
| CN108363619A (en) * | 2018-03-07 | 2018-08-03 | 深圳市酷开网络科技有限公司 | Service procedure control method, server and computer readable storage medium |
| CN110971687A (en) * | 2019-11-29 | 2020-04-07 | 浙江邦盛科技有限公司 | Rail transit flow data processing method |
| CN111522637A (en) * | 2020-04-14 | 2020-08-11 | 重庆邮电大学 | Storm task scheduling method based on cost benefit |
| CN111522637B (en) * | 2020-04-14 | 2024-03-29 | 深圳市凌晨知识产权运营有限公司 | Method for scheduling storm task based on cost effectiveness |
| CN113688115A (en) * | 2021-08-29 | 2021-11-23 | 中盾创新档案管理(北京)有限公司 | File big data distributed storage system based on Hadoop |
| CN113688115B (en) * | 2021-08-29 | 2024-02-20 | 中盾创新数字科技(北京)有限公司 | Archive big data distributed storage system based on Hadoop |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105824618A (en) | Real-time message processing method for Storm | |
| CN107943555B (en) | Big data storage and processing platform and big data processing method in cloud computing environment | |
| CN108958920B (en) | Distributed task scheduling method and system | |
| US10540351B2 (en) | Query dispatch and execution architecture | |
| US20160364269A1 (en) | Storage Resource Scheduling Method and Storage and Computing System | |
| US20150095917A1 (en) | Distributed uima cluster computing (ducc) facility | |
| US20100333084A1 (en) | Systems and methods for message-based installation management using message bus | |
| US20130145367A1 (en) | Virtual machine (vm) realm integration and management | |
| CN107291547A (en) | A kind of task scheduling processing method, apparatus and system | |
| CN113312165B (en) | Task processing method and device | |
| EP3103217B1 (en) | Monitoring system and monitoring method for software defined networks | |
| US9342291B1 (en) | Distributed update service | |
| CN105592127A (en) | Application management system for cloud computing environment | |
| US20220405122A1 (en) | Systems, methods, and apparatuses for processing routine interruption requests | |
| US10122602B1 (en) | Distributed system infrastructure testing | |
| CN110618821A (en) | Container cluster system based on Docker and rapid building method | |
| US7912956B1 (en) | Service level agreement based control of a distributed computing system | |
| US11086742B2 (en) | Task based service management platform | |
| CN113821322B (en) | Loosely coupled distributed workflow coordination system and method | |
| US9851980B1 (en) | Distributed update service enabling update requests | |
| WO2024139011A1 (en) | Information processing method | |
| US7043730B2 (en) | System and method for demand oriented network resource management | |
| JP2016506557A (en) | Transparent routing of job submissions between different environments | |
| CN107528709A (en) | A kind of configuration status backing method and device | |
| US20110246553A1 (en) | Validation of internal data in batch applications |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160803 |