CN104065540A - Data monitoring system and method - Google Patents
Data monitoring system and method Download PDFInfo
- Publication number
- CN104065540A CN104065540A CN201410331540.8A CN201410331540A CN104065540A CN 104065540 A CN104065540 A CN 104065540A CN 201410331540 A CN201410331540 A CN 201410331540A CN 104065540 A CN104065540 A CN 104065540A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- message queue
- processing module
- pending
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The embodiment of the invention provides a data monitoring system. The system comprises a data sending module, a message queue module and a data processing module. The data sending module is configured for sending data to be processed to the message queue module. The message queue module is configured for receiving and storing the data to be processed, wherein the data to be processed is sent by the data sending module, and the message queue module is used for sending the data to be processed to the data processing module. The data processing module is configured for processing the data according to a preset monitoring rule and outputting a processing result when the data to be processed are received, wherein the data to be processed are sent by the message queue module. Mass data are processed in real time, and the capacity for processing the real-time data of the system is improved obviously. In addition, the embodiment of the invention provides a data monitoring method.
Description
Technical field
Embodiments of the present invention relate to data processing field, and more specifically, embodiments of the present invention relate to a kind of data monitoring system and method.
Background technology
The embodiments of the present invention that be intended to for stating in claims this part provide background or context.Description is not herein because be included in just admit it is prior art in this part.
Along with Information technology development with rapid changepl. never-ending changes and improvements, information presents the expansion of explosion type, has produced a large amount of pending real time datas, such as network monitoring data, telecommunication call record, financial application data, Web daily record and clickstream data etc.These real time datas have that data volume is very big, formation speed is fast and the feature such as generating rate temporal evolution.How the in the situation that of resource-constrained, realize that these real time datas fast, are effectively treated as to a problem of needing solution badly.
Traditional relational database management system is efficiently processing aspect limited storage data, but inefficiency while processing a large amount of real time datas.
Summary of the invention
Therefore in the prior art, how processing efficiently magnanimity real time data, is very bothersome process.
For this reason, be starved of a kind of improved data monitoring system and method, to process efficiently magnanimity real time data.
In the present context, embodiments of the present invention expectation provides a kind of data monitoring system and method.
In the first aspect of embodiment of the present invention, a kind of system is provided, comprising: data transmission blocks, message queue module, data processing module, wherein:
Described data transmission blocks, is configured for to described message queue module and sends pending data;
Described message queue module, is configured for and receives and store the described pending data that described data transmission blocks sends, and, to described data processing module, send pending data;
Described data processing module, is configured for when receiving the pending data of described message queue module transmission, according to preset monitoring rules, described pending data is processed output result.
Preferably, described data transmission blocks is configured for asynchronously and sends pending data to described message queue module.
Preferably, described data processing module is configured for:
When described data processing module is during in idle condition, to described message queue module, send request of data and be used for asking pending data;
Described message queue block configuration is used for:
When receiving the request of data of described data processing module transmission, to the data processing module that sends request of data, send pending data.
Preferably, described message queue block configuration is used for:
According to the data subscription information of storage, to data processing module, send the pending data corresponding with described data subscription information.
Preferably, described data processing module comprises data aggregate unit and data collection module,
Described data aggregate cell location is used for: when described preset monitoring rules comprises monitored object and monitor control index, according to preset monitoring rules, pending data aggregate is processed into and take the key-value pair data that monitored object is value as key, monitor control index, and, data after polymerization is processed are carried out Hash operation, obtain the reference address of the data collection module corresponding with it, the data after described polymerization is processed send to relevant position;
Described data collection module is configured for: receive the data after described data aggregate unit polymerization is processed, to having the key-value pair data of same keys, merge and obtain amalgamation result.
Preferably, described data processing module comprises at least two data polymerized units and at least two data collection module.
Preferably, described system comprises at least two data transmission blocks; And/or
Described system comprises at least two message queue modules.
Preferably, described system also comprises:
Alarm module, is configured for the result after data processing module is processed is added up, when the statistics obtaining meets preset alert if, and output alarm information.
Preferably, described system also comprises:
Unified configuration module, is configured for described data processing module is unified to configuration.
In the second aspect of embodiment of the present invention, a kind of method is provided, be applied to data monitoring system, described system comprises data transmission blocks, message queue module and data processing module, described method comprises:
Described data transmission blocks sends pending data to described message queue module;
Described message queue module receives and stores the described pending data that described data transmission blocks sends, and, to described data processing module, send pending data;
Described data processing module receives the pending data that described message queue module sends, and according to preset monitoring rules, described pending data is processed output result.
The system and method that the embodiment of the present invention provides can be to real-time generation pending data, by data transmission blocks, message queue module, send to data processing module, by data processing module, realized the real-time processing of data, with output result, realize the real-time processing to mass data, significantly improved the disposal ability to real time data.
summary of the invention
The inventor finds, traditional relational database management system is efficiently processing aspect limited storage data, but inefficiency while processing a large amount of real time datas.
According to embodiment of the present invention, a kind of data monitoring system and method are provided, when needing deal with data, by data transmission blocks, to message queue module, send pending data, message queue module is for receiving and store pending data, and send pending data to data processing module, described data processing module, for when receiving the pending data of described message queue module transmission, is processed described pending data according to preset monitoring rules, output result.The system and method that the embodiment of the present invention provides can be to real-time generation pending data, by data transmission blocks, message queue module, send to data processing module, by data processing module, realized the real-time processing of data, with output result, realize the real-time processing to mass data, significantly improved the disposal ability of system to real time data.
After having introduced basic principle of the present invention, lower mask body is introduced various non-limiting execution mode of the present invention.
application scenarios overview
First with reference to figure 1, a kind of application scenarios of the embodiment of the present invention is described.Wherein, as shown in Figure 1, on line, data system 100 generates pending data, and data monitoring system 200 is for processing in real time the pending data that on described line, data system 100 generates.For example, on line, data system 100 is web Website servers, and it can the visit information of recording user to this web Website server.Certainly, it will be appreciated by persons skilled in the art that shown in Fig. 1 to be only schematically illustrating of a kind of application scenarios of the present invention, be not considered as the restriction to application scenarios of the present invention.
example devices
Below in conjunction with the application scenarios of Fig. 1, be described with reference to Figure 2 the data monitoring system according to exemplary embodiment of the invention.It should be noted that above-mentioned application scenarios is only to illustrate for the ease of understanding spirit of the present invention and principle, embodiments of the present invention are unrestricted in this regard.On the contrary, any scene that embodiments of the present invention can be applied to be suitable for.
A kind of data monitoring system 200, comprising: data transmission blocks 201, message queue module 202, data processing module 203, wherein:
Described data transmission blocks 201, is configured for to described message queue module and sends pending data.
Described message queue module 202, is configured for and receives and store the described pending data that described data transmission blocks sends, and, to described data processing module, send pending data.
Described data processing module 203, is configured for when receiving the pending data of described message queue module transmission, according to preset monitoring rules, described pending data is processed output result.
Wherein, data transmission blocks 201 is configured for when determining while there are pending data, to message queue module 202, sends pending data.Preferably, described data transmission blocks is configured for asynchronously and sends pending data to described message queue module.In order to realize data transmission blocks, to message queue, send data asynchronously, in data transmission blocks, can have a memory queue, for the pending data of data cached sending module.Like this, decoupling zero data system (sources of corresponding pending data) and message queue on the line of generated data, data transmission blocks sends pending data or message can not have influence on the data processing of data system on line.During specific implementation, described memory queue can adopt the producer/consumer mode treatment data, to guarantee the stable operation of data system on line.Further, data-interface and dispensing unit are set in data transmission blocks, described data-interface, for being connected with data-source device data, receives the pending data from data-source device (generating the equipment of pending data).Described data-interface, also for being connected with message queue module data, sends pending data to message queue module.Therefore, can to relevant parameter, be configured by dispensing unit, such as configuration, receive parameter, send parameter etc.According to this framework, various data-source device can access the data monitoring system that the embodiment of the present invention provides easily, thereby realize the real-time processing for the treatment of deal with data.
When specific implementation of the present invention, used message queue module to receive and store the described pending data that described data transmission blocks sends.Wherein, message queue (Message Queue), for those complicated data processing business are separated with online main business, provides abundant characteristic and extensibility.In a kind of possible implementation, can adopt ActiveMQ message queue, wherein, message queue module 202 is server, in data transmission blocks and two modules of data processing module, has respectively two clients, is used for issuing and subscribe message.During specific implementation, in order to improve the reliability of system, message queue module can adopt the mode of cluster, and for example system can comprise at least two message queue modules.When system comprises at least two message queue modules, can create SiteServer LBS, for realizing the load balancing of each message queue module, when any one message queue module breaks down or delay machine, can not affect the normal operation of whole message queue module cluster.
During specific implementation, message queue block configuration is for sending pending data to data processing module, described data processing module, be configured for when receiving the pending data of described message queue module transmission, according to preset monitoring rules, described pending data are processed output result.
Wherein, data processing module can initiatively pull the pending data of preserving in message queue module.When data processing module is during in idle condition, to described message queue module, send request of data and be used for asking pending data.Message queue module, when receiving the request of data of described data processing module transmission, sends pending data to the data processing module that sends request of data.That is to say, in this implementation, data processing module can, according to the processing condition data of self, choose the right moment and send pending data to the request of message queue module flexibly.Mode with respect to message queue module propelling data to data processing module, this mode is more flexible, because be initiatively pulls data when data processing module is idle, therefore can guarantee the data processing module deal with data of can having the ability, and the situation that the data that still cause to its propelling data while there will not be data processing module busy can not be processed in time.
Certainly, it will be understood by those skilled in the art that, in a kind of possible implementation, message queue module can also receive the data subscription information of data processing module, according to the data subscription information of storage, to data processing module, send the pending data corresponding with described data subscription information.Certainly, the data subscription information of message queue module stores can be also that system is pre-configured.Message queue module can send the pending data corresponding with data subscription information to data processing module in real time or periodically.
During specific implementation, data processing module is configured for when receiving the pending data of described message queue module transmission, according to preset monitoring rules, described pending data is processed output result.Wherein, described data processing module can comprise data aggregate unit and data collection module, described data aggregate cell location is used for: when described preset monitoring rules comprises monitored object and monitor control index, according to preset monitoring rules, pending data aggregate is processed into and take the key-value pair data that monitored object is value as key, monitor control index, and, data after polymerization is processed are carried out Hash operation, obtain the reference address of the data collection module corresponding with it, the data after described polymerization is processed send to relevant position; Described data collection module is configured for: receive the data after described data aggregate unit polymerization is processed, to having the key-value pair data of same keys, merge and obtain amalgamation result.Below in conjunction with can the give an example above-mentioned operative scenario of data of description polymerized unit and data collection module of Fig. 3.
During specific implementation, the data aggregate unit in data processing module, according to preset monitored object monitor control index and monitoring rules, becomes take to the pending data aggregate receiving the key-value pair data that monitored object is value as key, monitor control index.Wherein, monitored object can need to set in advance according to system, and monitor control index can be a certain attribute or the feature of monitored object.For monitored object, be for example that URL is example, monitor control index can be the access times of URL, the maximum access duration of URL, the average access duration of URL etc.Monitoring rules can be the monitor data in designated statistics predetermined amount of time, can be also monitor data of designated statistics predefined type etc.Monitoring rules can also comprise filtering rule, and for example by filtering rule, filtering out is obviously the network address of malice access, the correctness that reduces data statistics amount and improve data processing.For example, when URL comprise similar wave～～～while waiting spcial character, be likely malice network address, this class data filtering is fallen, do not process.
During specific implementation, the data after data collection module can be processed the polymerization of data polymerized unit are processed, and the key-value pair data with same keys are merged and obtain amalgamation result.The corresponding relation of data aggregate unit and data collection module can be one to one, can be also other corresponding relation.
Referring to Fig. 3, it schematically shows distributed real-time data supervisory control system according to another embodiment of the present invention.
In the distributed real-time data supervisory control system shown in Fig. 3, comprise three data transmission blocks 301, a message queue module 302, three data polymerized units 303, two data collection module 304 (being that data processing module comprises three data polymerized units 303 and two data collection module 304), two alarm modules 305, and a unified configuration module (not shown).Certainly, it will be understood by those of skill in the art that module in the distributed real-time data supervisory control system shown in Fig. 3 and the number of unit are schematic rather than restrictive.
It should be noted that, in this embodiment, inline system (or being called online application system, online data system) generates pending data, data transmission blocks can comprise a plurality of nodes, these a plurality of nodes can be positioned at same inline system, also can be positioned on different inline systems.Data transmission blocks is responsible for to message queue module asynchronous transmission data.When being sent, data do not rely on the stability of message queue module, therefore adopt in embodiments of the present invention asynchronous mode, particularly, in data transmission blocks, increase the stable operation that pattern that memory queue adopts producer consumer guarantees inline system.Owing to introducing the mechanism of buffer queue (memory queue) in data transmission blocks, decoupling zero the dependence of online application system and message queue module, avoided the impact on online application system in situation that goes wrong of message queue module.
In this embodiment, message queue module can be also distributed, can comprise a plurality of message queue submodules, for storing from separate sources or dissimilar pending data.Thereby make message queue module can store more data, and configuration is more flexible, a message queue submodule breaks down, and can not affect the operation of whole message queue module.For example, when one of them message queue submodule is delayed machine, can be rejected; Or, can increase a message queue submodule newly for substituting the message queue submodule breaking down, thereby guarantee the reliability service of message queue module.
Describe for example the operation principle of data aggregate unit and data collection module below.Suppose that preset monitored object is URL (Uniform Resource Locator, URL(uniform resource locator)), the access times that monitor control index is URL.
First, the pending data that receive from message queue module 302 are divided into 3 parts, and each in 3 data polymerized units is processed the pending data of corresponding portion.Suppose the data that comprise URL1, URL2 and URL3 in pending data, preset monitored object is URL1 and URL2.Suppose in 3 data polymerized units, the pending packet that data aggregate unit 1 is processed is the data of URL1 and URL3 containing reference address, the pending data that data aggregate unit 2 is processed comprise that reference address is the data of URL1 and URL2, and the pending data that data aggregate unit 3 is processed comprise that reference address is the data of URL2 and URL3.At this moment, after these data are processed in data aggregate unit 1, obtain the key-value pair that key is URL1; After processing these data, data aggregate unit 2 obtains the key-value pair that key is URL1 and URL2; After processing these data, data aggregate unit 3 obtains the key-value pair that key is URL2.The access times that the monitor control index of take is URL are example, and the result that data aggregate unit 1 carries out polymerization to be produced after processing is exactly centre key-value pair, for example (URL1:900), wherein 900 access times that represent URL.The result that data aggregate unit 2 carries out producing after polymerization is processed is exactly middle key-value pair (URL1:400), (URL2:600).The result that data aggregate unit 3 carries out producing after polymerization is processed is exactly middle key-value pair (URL2:1000).At this moment, each data aggregate unit can pass to same data collection module by the identical value of middle key assignments centering key producing.During specific implementation, each data aggregate unit carries out Hash operation according to the key key in key-value pair, thereby (for example obtain the reference address of data collection module, certain in data aggregate unit is specific to buffer memory or the memory address of a data collection module), then corresponding key-value pair is sent to respective cache or memory address, thereby corresponding data collector unit can, after a while by access respective cache or memory address, obtain corresponding key-value pair.For example, the key-value pair (URL1:900) that data aggregate unit 1 can be URL1 key is sent to buffer memory or the memory address R1 specific to data collection module 1.The key-value pair (URL1:400) that data aggregate unit 2 can be URL1 key is sent to buffer memory or the memory address R1 specific to data collection module 1, and the key-value pair that is URL2 key (URL2:600) is sent to specific to the buffer memory of data collection module 2 or memory address R2.The key-value pair (URL2:1000) that data aggregate unit 3 can be URL2 key is sent to buffer memory or the memory address R2 specific to data collection module 2.Like this, the key-value pair with same keys key=URL1 is finally all delivered to identical data collection module 1, by data collection module 1, received the value (900,400) of same key key (URL1), then this class value is merged to produce amalgamation result, amalgamation result is (URL1:1300).That is to say, by the processing of data aggregate unit and data collection module, the total access times that count key and be URL1 from numerous URL are 1300.Similarly, the key-value pair with same keys key=URL2 is finally all delivered to identical data collection module 2, by data collection module 2, received the value (600,1000) of same key key (URL2), then this class value is merged to produce amalgamation result, amalgamation result is (URL2:1600).That is to say, by the processing of data aggregate unit and data collection module, the total access times that count key and be URL2 from numerous URL are 1600.It should be noted that, these are only exemplary illustration, be not considered as limitation of the present invention.
In the present embodiment, described system can also comprise unified configuration module (Fig. 3 is not shown), is configured for described data processing module is unified to configuration.Owing to adopting distributed technology, data aggregate unit and data collection module have a plurality of, so very inconvenient if carry out configuration file deployment meeting on each unit.Therefore utilize in embodiments of the present invention unified configuration module to carry out unified management configuration to data polymerized unit and data collection module.For example, when the machine loading at data aggregate unit or data collection module place is higher, can register immediately a new machine becomes new data collection module or data aggregate unit.The machine if the machine at data aggregate unit or data collection module place is delayed, also can reject the machine breaking down rapidly, does not affect the operation of whole system.Again for example, by unified configuration module, the data from message queue module are divided into the umber corresponding with the number of data aggregate unit.Again for example, by unified configuration module, determine monitoring rules, monitored object and monitor control index, concrete hash function etc.
Further, the system of the embodiment of the present invention can also comprise: alarm module, and be configured for the result after data processing module is processed is added up, when the statistics obtaining meets preset alert if, output alarm information.Wherein, alarm module is analyzed according to the result after data processing module statistics.Alarm rule is configurable, supports the configuration item of a plurality of dimension many levels, and the mechanism such as mail warning, SMS alarm are provided.
In this embodiment of the present invention, data aggregate unit and data collection module adopt distributed design, and data aggregate unit obtains the reference address of data collection module by treating the Hash of monitored object.The unlimited horizontal extension of system like this, and do not need whole framework to do too much adjustment.Each data aggregate unit and data collection module are worked concurrently, have realized the real-time processing to mass data, have significantly improved the disposal ability to real time data.And because data aggregate unit and data collection module adopt distributed design, the fault-resistant ability of whole system increases substantially, and there will not be the bottleneck there will be in centralized design.
illustrative methods
After having introduced the equipment of exemplary embodiment of the invention, next, with reference to 4 pairs of exemplary embodiment of the invention of figure, for the method for data monitoring, be introduced.
A method, is applied to data monitoring system, and described system comprises data transmission blocks, message queue module and data processing module, and described method comprises:
S401, data transmission blocks sends pending data to described message queue module.
S402, described message queue module receives and stores the described pending data that described data transmission blocks sends, and, to described data processing module, send pending data.
S403, described data processing module receives the pending data that described message queue module sends, and according to preset monitoring rules, described pending data is processed output result.
Wherein, described data transmission blocks sends pending data to described message queue module asynchronously.
Further, described method also comprises:
When described data processing module is during in idle condition, to described message queue module, send request of data and be used for asking pending data;
Message queue module sends pending data to described data processing module and comprises:
When receiving the request of data of described data processing module transmission, to the data processing module that sends request of data, send pending data.
Wherein, described message queue module sends pending data to described data processing module and comprises:
According to the data subscription information of storage, to data processing module, send the pending data corresponding with described data subscription message.
Wherein, when described data processing module comprises data aggregate unit and data collection module, the pending data that the described message queue module of described reception sends, process described pending data according to preset monitoring rules, and output result comprises:
When described preset monitoring rules comprises monitored object and monitor control index, described data aggregate unit is processed into pending data aggregate to take monitored object as key according to preset monitoring rules, monitor control index is the key-value pair data of value, and, data after polymerization is processed are carried out Hash operation, obtain the reference address of the data collection module corresponding with it, the data after described polymerization is processed send to relevant position;
Described data collection module receives the data after described data aggregate unit polymerization is processed, and to having the key-value pair data of same keys, merges and obtains amalgamation result.
Further, described method also comprises:
Result after data processing module is processed is added up, when the statistics obtaining meets preset alert if, output alarm information.
Further, in the situation that described data processing module comprises a plurality of data aggregates unit and data collection module, described method also comprises:
Described data processing module is unified to arrange.
It should be noted that, above apparatus embodiments has been carried out at length introducing, embodiment of the method can realize with reference to apparatus embodiments.
Although it should be noted that some devices or the sub-device of having mentioned data monitoring system in above-detailed, this division is only exemplary not enforceable.In fact, according to the embodiment of the present invention, the feature of above-described two or more devices and function can be specialized in a device.Otherwise, the feature of an above-described device and function can Further Division for to be specialized by a plurality of devices.
In addition, although described in the accompanying drawings the operation of the inventive method with particular order,, this not requires or hint must be carried out these operations according to this particular order, or the operation shown in must carrying out all could realize the result of expectation.Additionally or alternatively, can omit some step, a plurality of steps be merged into a step and carry out, and/or a step is decomposed into a plurality of steps carries out.
Although described spirit of the present invention and principle with reference to some embodiments, but should be appreciated that, the present invention is not limited to disclosed embodiment, the division of each side is not meant that to the feature in these aspects can not combine to be benefited yet, and this division is only the convenience in order to explain.The present invention is intended to contain interior included various modifications and the equivalent arrangements of spirit and scope of claims.
Accompanying drawing explanation
By reference to accompanying drawing, read detailed description below, above-mentioned and other objects of exemplary embodiment of the invention, the feature and advantage easy to understand that will become.In the accompanying drawings, in exemplary and nonrestrictive mode, show some execution modes of the present invention, wherein:
Fig. 1 schematically shows a kind of application scenarios of the embodiment of the present invention;
Fig. 2 schematically shows a kind of realization according to the data monitoring system of embodiment of the present invention;
Fig. 3 schematically shows data monitoring system according to another embodiment of the present invention;
Fig. 4 schematically shows the data monitoring method according to further embodiment of this invention.
In the accompanying drawings, identical or corresponding label represents identical or corresponding part.
Embodiment
Below with reference to some illustrative embodiments, principle of the present invention and spirit are described.Should be appreciated that providing these execution modes is only used to make those skilled in the art can understand better and then realize the present invention, and not limit the scope of the invention by any way.
One skilled in the art will appreciate that embodiments of the present invention can be implemented as a kind of system, device, equipment, method or computer program.Therefore, the disclosure can specific implementation be following form, that is: hardware, software (comprising firmware, resident software, microcode etc.), or the form of hardware and software combination completely completely.
According to the embodiment of the present invention, a kind of data monitoring system and method have been proposed.
In this article, it will be appreciated that, the container of message preserved in related term " message queue " for being illustrated in the transmitting procedure of message.Usually, " message " is the data unit transmitting at two intercomputers.The Main Function of " message queue " is the transmission that route is provided and guarantees message.For example, while sending message, recipient is unavailable, and message queue meeting reservation message, until can successfully transmit it.
In addition, any number of elements in accompanying drawing is all unrestricted for example, and any name is all only for distinguishing, and does not have any limitation.
Below with reference to some representative embodiments of the present invention, explain in detail principle of the present invention and spirit.
Claims (10)
1. a system, comprises data transmission blocks, message queue module, data processing module, wherein:
Described data transmission blocks, is configured for to described message queue module and sends pending data;
Described message queue module, is configured for and receives and store the described pending data that described data transmission blocks sends, and, to described data processing module, send pending data;
Described data processing module, is configured for when receiving the pending data of described message queue module transmission, according to preset monitoring rules, described pending data is processed output result.
2. system according to claim 1, wherein:
Described data transmission blocks is configured for asynchronously and sends pending data to described message queue module.
3. system according to claim 1, wherein:
Described data processing module is configured for:
When described data processing module is during in idle condition, to described message queue module, send request of data and be used for asking pending data;
Described message queue block configuration is used for:
When receiving the request of data of described data processing module transmission, to the data processing module that sends request of data, send pending data.
4. system according to claim 1, wherein:
Described message queue block configuration is used for:
According to the data subscription information of storage, to data processing module, send the pending data corresponding with described data subscription information.
5. system according to claim 1, wherein:
Described data processing module comprises data aggregate unit and data collection module,
Described data aggregate cell location is used for: when described preset monitoring rules comprises monitored object and monitor control index, according to preset monitoring rules, pending data aggregate is processed into and take the key-value pair data that monitored object is value as key, monitor control index, and, data after polymerization is processed are carried out Hash operation, obtain the reference address of the data collection module corresponding with it, the data after described polymerization is processed send to relevant position;
Described data collection module is configured for: receive the data after described data aggregate unit polymerization is processed, to having the key-value pair data of same keys, merge and obtain amalgamation result.
6. system according to claim 5, wherein, described data processing module comprises at least two data polymerized units and at least two data collection module.
7. system according to claim 6, also comprises:
Unified configuration module, is configured for described data processing module is unified to configuration.
8. system according to claim 1, wherein:
Described system comprises at least two data transmission blocks; And/or
Described system comprises at least two message queue modules.
9. system according to claim 1, also comprises:
Alarm module, is configured for the result after data processing module is processed is added up, when the statistics obtaining meets preset alert if, and output alarm information.
10. a method, is applied to data monitoring system, and described system comprises data transmission blocks, message queue module and data processing module, and described method comprises:
Described data transmission blocks sends pending data to described message queue module;
Described message queue module receives and stores the described pending data that described data transmission blocks sends, and, to described data processing module, send pending data;
Described data processing module receives the pending data that described message queue module sends, and according to preset monitoring rules, described pending data is processed output result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410331540.8A CN104065540A (en) | 2014-07-11 | 2014-07-11 | Data monitoring system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410331540.8A CN104065540A (en) | 2014-07-11 | 2014-07-11 | Data monitoring system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104065540A true CN104065540A (en) | 2014-09-24 |
Family
ID=51553081
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410331540.8A Pending CN104065540A (en) | 2014-07-11 | 2014-07-11 | Data monitoring system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104065540A (en) |
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104579789A (en) * | 2015-01-23 | 2015-04-29 | 广东能龙教育股份有限公司 | Massive user behavior data acquisition method and system based on message queue |
| CN104796456A (en) * | 2015-03-17 | 2015-07-22 | 青岛海尔智能家电科技有限公司 | Messaging processing method and device |
| CN105306173A (en) * | 2015-11-17 | 2016-02-03 | 天泽信息产业股份有限公司 | Adaptive Internet of vehicles terminal access method |
| CN105306300A (en) * | 2015-11-03 | 2016-02-03 | 北京京东尚科信息技术有限公司 | Real-time network cluster monitoring method and system |
| CN105490835A (en) * | 2015-11-20 | 2016-04-13 | 北京百度网讯科技有限公司 | Information monitoring method and device |
| CN105591877A (en) * | 2014-10-23 | 2016-05-18 | 阿里巴巴集团控股有限公司 | Method, device and system for service message transmission |
| CN105843597A (en) * | 2015-01-16 | 2016-08-10 | 阿里巴巴集团控股有限公司 | Providing method and device for establishment information of tasks |
| CN106021061A (en) * | 2016-04-29 | 2016-10-12 | 北京奇虎科技有限公司 | Method, device and system for alarm information processing |
| CN106407391A (en) * | 2016-09-19 | 2017-02-15 | 北京集奥聚合科技有限公司 | A data quality monitoring method and system |
| CN107665139A (en) * | 2016-07-29 | 2018-02-06 | 北京新唐思创教育科技有限公司 | The implementation method and device drawn for the real time bidirectional in online teaching |
| CN108616459A (en) * | 2016-11-30 | 2018-10-02 | 天津易遨在线科技有限公司 | A kind of Message Queuing system that can promote message transmission speed |
| CN108874540A (en) * | 2018-06-04 | 2018-11-23 | 北京云鸟科技有限公司 | A kind of request processing method and system of the VRP cloud service system towards CPU intensive type |
| CN110297739A (en) * | 2019-06-04 | 2019-10-01 | 平安科技（深圳）有限公司 | The data monitoring method and relevant device of platform are administered based on data |
| CN110519217A (en) * | 2019-07-05 | 2019-11-29 | 中国平安人寿保险股份有限公司 | Across company-data transmission method, device, computer equipment and storage medium |
| CN111369007A (en) * | 2020-03-02 | 2020-07-03 | 第四范式（北京）技术有限公司 | Method and device for enabling artificial intelligence model to be online |
| CN112073506A (en) * | 2020-09-04 | 2020-12-11 | 哈尔滨工业大学 | IPv6 and message queue-based complex electromagnetic data acquisition method |
| CN114896058A (en) * | 2022-04-27 | 2022-08-12 | 南京鼎华智能系统有限公司 | Dispatching system and dispatching method based on memory operation |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103024014A (en) * | 2012-12-03 | 2013-04-03 | 北京京东世纪贸易有限公司 | Method and system for distributing and processing mass data passing through message queues |
| CN103150324A (en) * | 2012-12-26 | 2013-06-12 | 人民搜索网络股份公司 | Chained processing-based data collecting system and method |
| CN103914531A (en) * | 2014-03-31 | 2014-07-09 | 百度在线网络技术（北京）有限公司 | Method and device for processing data |
-
2014
- 2014-07-11 CN CN201410331540.8A patent/CN104065540A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103024014A (en) * | 2012-12-03 | 2013-04-03 | 北京京东世纪贸易有限公司 | Method and system for distributing and processing mass data passing through message queues |
| CN103150324A (en) * | 2012-12-26 | 2013-06-12 | 人民搜索网络股份公司 | Chained processing-based data collecting system and method |
| CN103914531A (en) * | 2014-03-31 | 2014-07-09 | 百度在线网络技术（北京）有限公司 | Method and device for processing data |
Cited By (23)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105591877A (en) * | 2014-10-23 | 2016-05-18 | 阿里巴巴集团控股有限公司 | Method, device and system for service message transmission |
| CN105843597A (en) * | 2015-01-16 | 2016-08-10 | 阿里巴巴集团控股有限公司 | Providing method and device for establishment information of tasks |
| CN104579789A (en) * | 2015-01-23 | 2015-04-29 | 广东能龙教育股份有限公司 | Massive user behavior data acquisition method and system based on message queue |
| CN104796456A (en) * | 2015-03-17 | 2015-07-22 | 青岛海尔智能家电科技有限公司 | Messaging processing method and device |
| CN104796456B (en) * | 2015-03-17 | 2019-06-28 | 青岛海尔智能家电科技有限公司 | A kind of message treatment method and device |
| CN105306300A (en) * | 2015-11-03 | 2016-02-03 | 北京京东尚科信息技术有限公司 | Real-time network cluster monitoring method and system |
| CN105306173A (en) * | 2015-11-17 | 2016-02-03 | 天泽信息产业股份有限公司 | Adaptive Internet of vehicles terminal access method |
| CN105490835A (en) * | 2015-11-20 | 2016-04-13 | 北京百度网讯科技有限公司 | Information monitoring method and device |
| CN105490835B (en) * | 2015-11-20 | 2019-03-01 | 北京百度网讯科技有限公司 | Information monitoring method and device |
| CN106021061A (en) * | 2016-04-29 | 2016-10-12 | 北京奇虎科技有限公司 | Method, device and system for alarm information processing |
| CN106021061B (en) * | 2016-04-29 | 2019-05-28 | 北京奇虎科技有限公司 | Alarm information processing method, apparatus and system |
| CN107665139A (en) * | 2016-07-29 | 2018-02-06 | 北京新唐思创教育科技有限公司 | The implementation method and device drawn for the real time bidirectional in online teaching |
| CN106407391A (en) * | 2016-09-19 | 2017-02-15 | 北京集奥聚合科技有限公司 | A data quality monitoring method and system |
| CN106407391B (en) * | 2016-09-19 | 2020-01-07 | 北京集奥聚合科技有限公司 | Data quality monitoring method and system |
| CN108616459A (en) * | 2016-11-30 | 2018-10-02 | 天津易遨在线科技有限公司 | A kind of Message Queuing system that can promote message transmission speed |
| CN108874540A (en) * | 2018-06-04 | 2018-11-23 | 北京云鸟科技有限公司 | A kind of request processing method and system of the VRP cloud service system towards CPU intensive type |
| CN110297739A (en) * | 2019-06-04 | 2019-10-01 | 平安科技（深圳）有限公司 | The data monitoring method and relevant device of platform are administered based on data |
| CN110519217A (en) * | 2019-07-05 | 2019-11-29 | 中国平安人寿保险股份有限公司 | Across company-data transmission method, device, computer equipment and storage medium |
| CN111369007A (en) * | 2020-03-02 | 2020-07-03 | 第四范式（北京）技术有限公司 | Method and device for enabling artificial intelligence model to be online |
| CN111369007B (en) * | 2020-03-02 | 2024-02-13 | 第四范式（北京）技术有限公司 | Method and device for online artificial intelligent model |
| CN112073506A (en) * | 2020-09-04 | 2020-12-11 | 哈尔滨工业大学 | IPv6 and message queue-based complex electromagnetic data acquisition method |
| CN114896058A (en) * | 2022-04-27 | 2022-08-12 | 南京鼎华智能系统有限公司 | Dispatching system and dispatching method based on memory operation |
| CN114896058B (en) * | 2022-04-27 | 2023-09-22 | 南京鼎华智能系统有限公司 | Dispatching system and dispatching method based on memory operation |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104065540A (en) | Data monitoring system and method | |
| EP4120166A1 (en) | Blockchain message processing method and apparatus, computer and readable storage medium | |
| CN102469033B (en) | Message subscription system and message sending method | |
| CN102479207B (en) | Information search method, system and device | |
| CN104754036A (en) | Message processing system and processing method based on kafka | |
| CN106209353A (en) | Key management method and system thereof | |
| CN110546923A (en) | selective distribution of messages in a scalable real-time messaging system | |
| CN111061804A (en) | Asynchronous data processing method, device, equipment and storage medium based on big data | |
| CN111770002B (en) | Test data forwarding control method and device, readable storage medium and electronic equipment | |
| CN106130960B (en) | Judgement system, load dispatching method and the device of steal-number behavior | |
| CN111782620A (en) | Credit link automatic tracking platform and method thereof | |
| CN105306300A (en) | Real-time network cluster monitoring method and system | |
| CN111641524A (en) | Monitoring data processing method, device, equipment and storage medium | |
| CN108076103A (en) | A kind of message treatment method, system and message producer's cluster | |
| CN104916095A (en) | Multi-point monitoring alarm method and system | |
| CN108933966B (en) | Request message processing method, device and system, terminal and storage medium | |
| CN107018170A (en) | A kind of method of data synchronization and equipment | |
| CN106686040A (en) | Message processing method and message processing device | |
| CN108134745A (en) | Distribution type data collection method and device | |
| CN108093047B (en) | Data sending method and device, electronic equipment and middleware system | |
| CN110798495B (en) | Method and server for end-to-end message push in cluster architecture mode | |
| CN112527530B (en) | Message processing method, device, apparatus, storage medium and computer program product | |
| CN113342503A (en) | Real-time progress feedback method, device, equipment and storage medium | |
| CN111651425A (en) | Data extraction method and device for data mart, terminal and storage medium | |
| CN116248304A (en) | Cloud service message transfer monitoring system for high-performance computing and application method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20140924 |
|
| RJ01 | Rejection of invention patent application after publication |