CN108733534A - monitoring system, data processing method and device - Google Patents

monitoring system, data processing method and device Download PDF

Info

Publication number
CN108733534A
CN108733534A CN201710269398.2A CN201710269398A CN108733534A CN 108733534 A CN108733534 A CN 108733534A CN 201710269398 A CN201710269398 A CN 201710269398A CN 108733534 A CN108733534 A CN 108733534A
Authority
CN
China
Prior art keywords
data
mentioned
storage
compressed
preset
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201710269398.2A
Other languages
Chinese (zh)
Inventor
梁增海
吴毓群
陈朝晖
陈武材
何珂程
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Jingdong Century Trading Co Ltd
Beijing Jingdong Shangke Information Technology Co Ltd
Original Assignee
Beijing Jingdong Century Trading Co Ltd
Beijing Jingdong Shangke Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Jingdong Century Trading Co Ltd, Beijing Jingdong Shangke Information Technology Co Ltd filed Critical Beijing Jingdong Century Trading Co Ltd
Priority to CN201710269398.2A priority Critical patent/CN108733534A/en
Publication of CN108733534A publication Critical patent/CN108733534A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/30Monitoring
    • G06F11/3055Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/30Monitoring
    • G06F11/3003Monitoring arrangements specially adapted to the computing system or computing system component being monitored
    • G06F11/3006Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems

Abstract

This application discloses monitoring system, data processing method and device.One specific implementation mode of the system includes:Data collection terminal, data compression end and the data storage end;The data collection terminal carries out data encapsulation to the monitoring data and generates the first data, which is written preset memory locations for acquiring the monitoring data under preset monitoring parameter from monitored server;The first data extracted are compressed based on the preset monitoring parameter, and compressed data are periodically sent to the data storage end for extracting the first data from the preset memory locations at the data compression end;The data storage end, for being stored to the compressed data received.The embodiment, which realizes, is imbued with targetedly data storage.

Description

Monitoring system, data processing method and device
Technical field
This application involves field of computer technology, and in particular to Internet technical field more particularly to monitoring system, data Treating method and apparatus.
Background technology
With the continuous development of Internet technology, the machine included by for example large-scale distributed system of existing system is many It is more, and business is complicated, and visit capacity is usually bigger.Especially after containerization technique is widely used, machine quantity is even more Sharp rise.For system (such as the distributed system of 7 × 24 hours continuous operations, in financial field of uninterrupted operation Trading server, the game server in internet gaming field, electric business field trading server cluster etc.) real-time fortune The monitoring of row state is extremely important.
Although existing monitoring system can realize the monitoring to monitored server, collected monitoring data is deposited Storage typically takes up larger memory space.
Invention content
The purpose of the application is to propose a kind of improved monitoring system, data processing method and device, more than solving The technical issues of background technology part is mentioned.
In a first aspect, the embodiment of the present application provides a kind of monitoring system, which includes:Data collection terminal, data pressure Contracting end and the data storage end;
Above-mentioned data collection terminal, for acquiring the monitoring data under preset monitoring parameter from monitored server, to above-mentioned Monitoring data carries out data encapsulation and generates the first data, and preset memory locations are written in above-mentioned first data;Above-mentioned data compression End, for extracting the first data from above-mentioned preset memory locations, based on above-mentioned preset monitoring parameter to the first data for extracting It is compressed, and compressed data is periodically sent to above-mentioned the data storage end;Above-mentioned the data storage end, for pair The compressed data received are stored.
In some embodiments, above-mentioned preset monitoring parameter includes at least one of following:Take value, request results, wherein Above-mentioned time-consuming value refers to above-mentioned monitored server and carries out handling spent time, above-mentioned request results packet to the request received It includes successfully and fails.
In some embodiments, above-mentioned that the first data extracted are compressed based on above-mentioned preset monitoring parameter, packet It includes:If above-mentioned preset monitoring parameter includes taking value, value is taken for each included by the first data for extracting, Included time-consuming range, which is provided with, determines that this takes the target at value place and takes model in the preset time-consuming range set of count value It encloses, increases the count value that above-mentioned target takes range.
In some embodiments, above-mentioned that the first data extracted are compressed based on above-mentioned preset monitoring parameter, packet It includes:If above-mentioned preset monitoring parameter includes request results, each request included by the first data for extracting is tied Fruit determines that target identical with the request results is asked in the request results set that included request results are provided with count value It asks as a result, increasing the count value of above-mentioned destination request result.
In some embodiments, it is previously provided with storage region sequence and is used to indicate currently used target storage The storage region parameter in domain;And it is above-mentioned the first data extracted are compressed based on above-mentioned preset monitoring parameter, including: The meter that currently determined target takes range is recorded in the currently indicated target storage domain of parameter in the above storage area The count value of numerical value and/or identified destination request result.
In some embodiments, above-mentioned that compressed data are periodically sent to above-mentioned the data storage end, including: When arriving when send compressed data at the time of, determine in above-mentioned storage region sequence with the presence or absence of sequence inferior to current The storage region in target storage domain;It is deposited inferior to current target if so, above-mentioned storage region parameter is made to be directed toward said sequence The storage region in storage area domain;By in above-mentioned current target storage domain each count value and target corresponding with the count value It takes range or destination request result and is sent to above-mentioned the data storage end as compressed data, and empty above-mentioned current Count value in the domain of target storage.
In some embodiments, above-mentioned that compressed data are periodically sent to above-mentioned the data storage end, including: In response to there is no the storage region that sequence is inferior to current target storage domain, then making in the above-mentioned storage region sequence of determination It states storage region parameter and is directed toward the storage region for being located at above-mentioned storage region sequence header.
Second aspect, the embodiment of the present application provide a kind of data processing method, and this method includes:From preset memory locations Extract the first data, wherein above-mentioned first data are data collection terminals to the default monitoring parameter acquired from monitored server Under monitoring data carry out the data that are generated after data encapsulation;Based on above-mentioned default monitoring parameter to the first data for extracting It is compressed;Compressed data are periodically sent to the data storage end, so that above-mentioned the data storage end is to receiving Compressed data stored.
In some embodiments, above-mentioned preset monitoring parameter includes at least one of following:Take value, request results, wherein Above-mentioned time-consuming value refers to above-mentioned monitored server and carries out handling spent time, above-mentioned request results packet to the request received It includes successfully and fails.
In some embodiments, above-mentioned that the first data extracted are compressed based on above-mentioned default monitoring parameter, packet It includes:If above-mentioned preset monitoring parameter includes taking value, value is taken for each included by the first data for extracting, Included time-consuming range, which is provided with, determines that this takes the target at value place and takes model in the preset time-consuming range set of count value It encloses, increases the count value that above-mentioned target takes range.
In some embodiments, above-mentioned that the first data extracted are compressed based on above-mentioned default monitoring parameter, packet It includes:If above-mentioned preset monitoring parameter includes request results, each request included by the first data for extracting is tied Fruit determines that target identical with the request results is asked in the request results set that included request results are provided with count value It asks as a result, increasing the count value of above-mentioned destination request result.
In some embodiments, it is previously provided with storage region sequence and is used to indicate currently used target storage The storage region parameter in domain;And it is above-mentioned the first data extracted are compressed based on above-mentioned default monitoring parameter, including: The meter that currently determined target takes range is recorded in the currently indicated target storage domain of parameter in the above storage area The count value of numerical value and/or identified destination request result.
In some embodiments, above-mentioned that compressed data are periodically sent to the data storage end, including:Work as hair When arriving at the time of the warp let-off compressed data, determines in above-mentioned storage region sequence and be inferior to current target with the presence or absence of sequence The storage region of storage region;If so, so that above-mentioned storage region parameter is directed toward said sequence is inferior to current target storage The storage region in domain;By in above-mentioned current target storage domain each count value and target corresponding with the count value take Range or destination request result are sent to above-mentioned the data storage end as compressed data, and empty above-mentioned current target Count value in storage region.
In some embodiments, above-mentioned that compressed data are periodically sent to the data storage end, including:Response In determining in above-mentioned storage region sequence there is no the storage region that sequence is inferior to current target storage domain, then make above-mentioned deposit Storage area field parameter is directed toward the storage region positioned at above-mentioned storage region sequence header.
The third aspect, the embodiment of the present application provide a kind of data processing equipment, which includes:Data extracting unit, It is configured to extract the first data from preset memory locations, wherein above-mentioned first data are data collection terminals to from monitored clothes Monitoring data under the default monitoring parameter of business device acquisition carries out the data generated after data encapsulation;Compression unit, configuration are used In being compressed to the first data extracted based on above-mentioned default monitoring parameter;Transmission unit periodically will be configured to Compressed data are sent to the data storage end, so that above-mentioned the data storage end carries out the compressed data received Storage.
In some embodiments, above-mentioned preset monitoring parameter includes at least one of following:Take value, request results, wherein Above-mentioned time-consuming value refers to above-mentioned monitored server and carries out handling spent time, above-mentioned request results packet to the request received It includes successfully and fails.
In some embodiments, above-mentioned compression unit is further configured to:If above-mentioned preset monitoring parameter includes taking Value then takes value for each included by the first data for extracting, included time-consuming range is provided with count value Preset time-consuming range set in determine this take value where target take range, increase above-mentioned target take range counting Value.
In some embodiments, above-mentioned compression unit is further configured to:If above-mentioned preset monitoring parameter includes request As a result, being then provided with meter in included request results for each request results included by the first data for extracting Determine destination request identical with the request results as a result, increasing above-mentioned destination request result in the request results set of numerical value Count value.
In some embodiments, it is previously provided with storage region sequence and is used to indicate currently used target storage The storage region parameter in domain;And above-mentioned compression unit is further configured to:Parameter is currently indicated in the above storage area Target storage domain in record currently determined target take range count value and/or identified destination request result Count value.
In some embodiments, above-mentioned transmission unit is further configured to:At the time of sending compressed data When arrival, the storage region for being inferior to current target storage domain in above-mentioned storage region sequence with the presence or absence of sequence is determined;If It is that above-mentioned storage region parameter is then made to be directed toward the storage region that said sequence is inferior to current target storage domain;Work as above-mentioned Each count value and target corresponding with the count value in preceding target storage domain take range or destination request result is made It is sent to above-mentioned the data storage end for compressed data, and empties the count value in above-mentioned current target storage domain.
In some embodiments, above-mentioned transmission unit is further configured to:In response to the above-mentioned storage region sequence of determination In there is no the storage region that sequence is inferior to current target storage domain, then make above-mentioned storage region parameter be directed toward be located at it is above-mentioned The storage region of storage region sequence header.
Fourth aspect, the embodiment of the present application provide a kind of electronic equipment, which includes:One or more processing Device;Storage device, for storing one or more programs;When said one or multiple programs are by said one or multiple processors It executes so that said one or multiple processors realize the method as described in any realization method in second aspect.
5th aspect, the embodiment of the present application provide a kind of computer readable storage medium, are stored thereon with computer journey Sequence realizes the method as described in any realization method in second aspect when above procedure is executed by processor.
Monitoring system provided by the embodiments of the present application acquires preset monitoring from monitored server by data collection terminal and joins Monitoring data under several carries out data encapsulation to the monitoring data and generates the first data, which is written default storage Position.The first data are extracted from the preset memory locations by data compression end, based on the preset monitoring parameter to extracting First data are compressed, and compressed data are periodically sent to the data storage end.Pass through the data storage end Compressed data are stored.So as to efficiently use preset monitoring parameter, for different preset monitoring parameters, Different data compression algorithms may be used to compress the first data extracted, and then may be implemented to be imbued with targetedly Data store.
Data processing method and device provided by the embodiments of the present application, by extracting the first data from preset memory locations, To be compressed to the first data extracted based on default monitoring parameter, and periodically compressed data are sent To the data storage end, so that the data storage end stores the compressed data received.So as to effectively profit With preset monitoring parameter, for different preset monitoring parameters, may be used different data compression algorithms to extract One data are compressed, and then may be implemented to be imbued with targetedly data storage.
Description of the drawings
By reading a detailed description of non-restrictive embodiments in the light of the attached drawings below, the application's is other Feature, objects and advantages will become more apparent upon:
Fig. 1 is the exemplary architecture figure according to the monitoring system of the application;
Fig. 2 is the sequence diagram according to an embodiment of the monitoring system of the application;
Fig. 3 is the schematic diagram of an application scenarios corresponding with embodiment shown in Fig. 2;
Fig. 4 is the flow chart according to an embodiment of the data processing method of the application;
Fig. 5 is the structural schematic diagram according to an embodiment of the data processing equipment of the application;
Fig. 6 is adapted for the structural schematic diagram of the computer system of the electronic equipment for realizing the embodiment of the present application.
Specific implementation mode
The application is described in further detail with reference to the accompanying drawings and examples.It is understood that this place is retouched The specific embodiment stated is used only for explaining related invention, rather than the restriction to the invention.It also should be noted that in order to Convenient for description, is illustrated only in attached drawing and invent relevant part with related.
It should be noted that in the absence of conflict, the features in the embodiments and the embodiments of the present application can phase Mutually combination.The application is described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.
Fig. 1 shows monitoring system, the embodiment of data processing method or data processing equipment that can apply the application Exemplary system architecture 100.
As shown in Figure 1, system architecture 100 may include monitored server 101, and network 102,104,106, data acquisition End 103, data compression end 105 and the data storage end 107.Network 102 is in monitoring server 101 and data collection terminal 103 Between provide communication link medium.Network 104 between data collection terminal 103 and data compression end 105 providing communication The medium of link.Network 106 between data compression end 105 and the data storage end 107 provide communication link medium.Net Network 102,104,106 may include various connection types, such as wired, wireless communication link or fiber optic cables etc..
Monitored server 101 can be to provide the server of various services, such as from the terminal device (Fig. 1 connected In be not shown) receive different request (such as data inquiry request, data storage request, data computation requests etc.), can be with The request received is handled.
Data collection terminal 103 can be support the terminal of data acquisition function, server, or possess software and hardware resources into Journey.Data collection terminal 103 can from monitored server 101 acquire monitoring data, and to collected monitoring data at Reason.
Data compression end 105 can be support the terminal of data compression function, server, or possess software and hardware resources into Journey.It data compression end 105 can be to through data collection terminal 103, treated that data are compressed, and by compressed data It is sent to the data storage end 107.
The data storage end 107 can be support the terminal of data storage function, server, or possess software and hardware resources into Journey.The data storage end 107 can be stored to receiving compressed data, such as the data are stored to telecommunication The server (not shown in figure 1) of connection.
It should be noted that the data processing method that the embodiment of the present application is provided generally is executed by data compression end 105, Correspondingly, data processing equipment is generally positioned in data compression end 105.
It should be understood that monitored server, network, data collection terminal, data compression end and the data storage end in Fig. 1 Number is only schematical.According to needs are realized, can have any number of monitored server, network, data acquisition End, data compression end and the data storage end.
It should be pointed out that when data collection terminal 103, data compression end 105 and the data storage end 107 are to possess software and hardware When the process of resource, data collection terminal 103, data compression end 105 and the data storage end 107 can be arranged in same server In.
With continued reference to Fig. 2, Fig. 2 shows the sequence diagrams according to an embodiment of the monitoring system of the application.
The monitoring system of the present embodiment may include:Data collection terminal, data compression end and the data storage end;The data are adopted Collect end, for acquiring the monitoring data under preset monitoring parameter from monitored server, data encapsulation is carried out to the monitoring data The first data are generated, which is written into preset memory locations;The data compression end, for being carried from the preset memory locations The first data are taken, the first data extracted are compressed based on the preset monitoring parameter, and periodically will after compression Data be sent to the data storage end;The data storage end, for being stored to the compressed data received.
As shown in Fig. 2, in step 201, data collection terminal acquires the prison under preset monitoring parameter from monitored server Control data.
In the present embodiment, data collection terminal (such as data collection terminal 103 shown in FIG. 1) can in real time or periodically (such as every 5 seconds, 30 seconds, one minute etc.) are pre- from monitored server (such as monitored server 101 shown in FIG. 1) acquisition Set the monitoring data under monitoring parameter.Here, preset monitoring parameter can be artificially arranged, and preset monitoring parameter can be such as Including request amount, which can refer to request that above-mentioned monitored server receives within a predetermined period of time (such as data Inquiry request, data storage request etc.) quantity.Here, above-mentioned predetermined amount of time can be artificially arranged, can also be on State what data collection terminal was arranged automatically, the duration of above-mentioned predetermined amount of time can be changed, and the present embodiment is not to content in this respect Do any restriction.
Optionally, above-mentioned preset monitoring parameter can also include at least one of following:Take value, request results.Wherein, should Time-consuming value can refer to above-mentioned monitored server and carry out handling the spent time to the request received, which can be with Including success and failure.
In step 202, data collection terminal carries out data encapsulation to monitoring data and generates the first data, and the first data are write Enter preset memory locations.
In the present embodiment, above-mentioned data collection terminal can carry out collected monitoring data data encapsulation and generate first Data, and preset memory locations are written into first data.Wherein, which can be the message team being generated in advance Row (message queue is often referred to preserve the container of message in the transmission process of message), can also be shared drive (shared memory).Here, shared drive can refer in the computer system of multiprocessor, the large capacity that can be accessed by different CPU Memory.It should be pointed out that data encapsulation is often referred to business datum to be mapped in the payload of some tunneling, it is subsequently filled The packet header of corresponding agreement, forms the data packet of tunneling.Since data encapsulation is the known skill studied and applied extensively at present Art, details are not described herein.
In some optional realization methods of the present embodiment, it can pre-set and be useful in above-mentioned monitored server Handle the business process of request.When above-mentioned data collection terminal is the process supported data acquisition function, possess software and hardware resources When, which can be embedded in by the form of class libraries (Class Library) in above-mentioned business process, to realize to above-mentioned The acquisition of monitoring data under preset monitoring parameter.Obtained by the process can also will be encapsulated as Inter-Process Communication form through data The first data above-mentioned message queue or above-mentioned shared drive is written.Here, what class libraries was often referred to comprehensive object-oriented can Reuse types set.
In some optional realization methods of the present embodiment, above-mentioned data collection terminal can be also used for encapsulate through data The first data of gained are sent to the data processing server being connect with above-mentioned data collection terminal telecommunication, so that the data processing Server is further analyzed processing to the first data received.
In step 203, the first data are extracted in data compression end from preset memory locations, based on preset monitoring parameter to carrying The first data taken out are compressed.
In the present embodiment, data compression end (such as data compression end 105 shown in FIG. 1) can be from above-mentioned default storage The first data are extracted in position, and are compressed to the first data extracted based on above-mentioned preset monitoring parameter.Here, above-mentioned number According to the sequencing for the time that compression end can be written into according to the first data, successively from above-mentioned preset memory locations extraction first Data.It should be pointed out that above-mentioned data compression end can in real time or periodically (such as every 10 seconds etc.) are preset from above-mentioned Storage location extracts the first data.
In the present embodiment, if above-mentioned preset monitoring parameter includes request amount, above-mentioned data compression end can be pre-set There are preset value (such as 1000), above-mentioned data compression end that can subtract the request amount included by the first data extracted every time Above-mentioned preset value carries out compression institute using the parameter name of the difference of gained and preset monitoring parameter request amount as to first data The compressed data obtained.
It should be pointed out that above-mentioned data compression end is before compressing the first data extracted, it can be to this First data are decapsulated.Decapsulation is the inverse process of encapsulation, and decapsulation is often referred to dismantling protocol package, handles the letter in packet header Breath takes out the business information in payload.
In some optional realization methods of the present embodiment, if above-mentioned preset monitoring parameter includes taking value, for Each included by the first data extracted takes value, and above-mentioned data compression end can be arranged in included time-consuming range Have and determine that this takes the target at value place and takes range in the preset time-consuming range set of count value, increases the target and take range Count value.Here, the initial value that each in above-mentioned preset time-consuming range set takes the count value of range can be 0.Make For example, it is assumed that above-mentioned time-consuming range set include take range " 0ms~10ms ", " 11ms~20ms ", ">20ms ", the consumption When range count value be 0, wherein ms indicate millisecond, full name in English millisecond.It is assumed that currently extract One data A includes taking value (unit for taking value is millisecond) 10,24,25,5, and above-mentioned data compression end can determine time-consuming value 5, it is " 0ms~10ms " that the target where 10, which takes range, above-mentioned data compression end target can be taken range " 0ms~ The count value of 10ms " increases to 2.Above-mentioned data compression end can determine that the target where taking value 24,25 takes range and is ">20ms ", above-mentioned data compression end target can be taken range ">The count value of 20ms " increases to 2.Above-mentioned data compression end The first data B can also be extracted from above-mentioned preset memory locations, it is assumed that the first data B includes taking value 10,15,21, above-mentioned The count value that target can be taken range " 0ms~10ms " by data compression end increases to 3 from 2, and target is taken range " 11ms The count value of~20ms " increases to 1 from 0, by target take range ">The count value of 20ms " increases to 3 from 2.
It is right if above-mentioned preset monitoring parameter includes request results in some optional realization methods of the present embodiment In each request results included by the first data extracted, above-mentioned data compression end can be in included request results It is provided in the request results set of count value and determines destination request identical with the request results as a result, increasing the destination request As a result count value.Here, the initial value of the count value of each request results in above-mentioned request results set can be 0.Make For example, request results set A includes request results " success " and " failure ", the count value of request results " success " and " failure " It is 0.It is assumed that the first data extracted include request results " success ", " success ", " failure ", then above-mentioned data compression end can With determined in request results set A with request results " success ", the destination request result " success " of " success " all same, on 2 can be increased to by the count value of destination request result " success " by stating data compression end.Above-mentioned data compression end can ask Determined in results set A with the identical destination request result " failure " of request results " failure ", above-mentioned data compression end can be with The count value of destination request result " failure " is increased to 1.
It is pointed out that in the time-consuming range and above-mentioned preset request results set in above-mentioned preset time-consuming range set Request results can be changed, the present embodiment does not do any restriction to content in this respect.In addition, above-mentioned preset time-consuming model Above-mentioned data compression end can be stored in advance in by enclosing set and above-mentioned preset request results set.It should be noted that through pressure After data after contracting are issued, the count value of the time-consuming range included by above-mentioned preset time-consuming range set and above-mentioned preset ask Ask the count value of the request results included by results set that can be initialised.
It should be noted that if above-mentioned preset monitoring parameter includes request amount and at least one of following:Take value, request knot Fruit, above-mentioned data compression end can by each time-consuming value included by the first data to extracting and/or request results into Row compression is to realize the compression to first data.Because determining that each target takes the counting of range for first data After the count value of value or each target result of calculation, it is added by the count value to the time-consuming range of each target or right The count value of each target result of calculation is added, and the request amount included by first data can be calculated.
In some optional realization methods of the present embodiment, it can be previously provided in the memory at above-mentioned data compression end Storage region sequence and the storage region parameter for being used to indicate currently used target storage domain, wherein the storage region Each storage region in sequence can be provided with mark.When above-mentioned data compression end to extract the first data compression when, Above-mentioned data compression end can record in the currently indicated target storage domain of parameter currently determined in the above storage area Target take range count value and/or identified destination request result count value.As an example, above-mentioned storage region Storage region included by sequence is followed successively by storage region A and storage region B, it is assumed that above-mentioned storage region parameter is currently indicated Target storage domain be storage region B, above-mentioned data compression end can record currently determined target consumption in storage region B When the count value of the range and/or count value of identified destination request result.
In some optional realization methods of the present embodiment, when above-mentioned data collection terminal and above-mentioned data compression end are When possessing the process of software and hardware resources, above-mentioned data collection terminal can pass through TCP (Transmission Control Protocol, transmission control protocol) or UDP (User Datagram Protocol, User Datagram Protocol) agreement transmission number According to extremely above-mentioned data compression end.
In step 204, compressed data are periodically sent to the data storage end by data compression end.
In the present embodiment, above-mentioned data compression end can periodically (such as every 30 seconds, one minute etc.) will be through pressure Data after contracting are sent to the data storage end (such as the data storage end 107 shown in FIG. 1).As an example, assuming above-mentioned preset Monitoring parameter includes request amount, and when (time point) arrives when send compressed data at the time of, above-mentioned data compression end can (will not be sent after compression including one on this between the moment and a upper time point for sending compressed data Data time point) calculated each difference (request amount included by the first data extracted subtracts above-mentioned pre- Set the difference obtained by value) and the parameter name of preset monitoring parameter request amount be sent to above-mentioned data as compressed data and deposit Chu Duan.Here, being at the time of sending compressed data for the first time can be according to pre-set initial time and interval duration (such as 30 seconds, one minute etc.) are calculated.It is non-be at the time of send compressed data for the first time can be according to the moment Upper one time point for sending compressed data and above-mentioned interval duration calculation obtain.It should be noted that above-mentioned Interval duration can be changed according to actual needs, and the present embodiment does not do any restriction to content in this respect.
In some optional realization methods of the present embodiment, if above-mentioned preset monitoring parameter includes taking value and/or asking It asks as a result, when arriving when send compressed data at the time of, above-mentioned data compression end can determine above-mentioned storage region sequence The storage region in current target storage domain is inferior in row with the presence or absence of sequence;If so, above-mentioned data compression end can incite somebody to action Above-mentioned storage region parameter is directed toward storage region of the said sequence inferior to current target storage domain (such as by above-mentioned memory block The parameter value of field parameter is revised as mark of the said sequence inferior to the storage region in current target storage domain);If above-mentioned number According to compression end determine in above-mentioned storage region sequence there is no sequence be inferior to current target storage domain storage region, then on Above-mentioned storage region parameter can be directed toward the storage region for being located at above-mentioned storage region sequence header by stating data compression end;It is above-mentioned It data compression end can be by each count value and target corresponding with count value consumption in above-mentioned current target storage domain When range or destination request result be sent to above-mentioned the data storage end as compressed data, and empty above-mentioned current mesh Mark the count value in storage region.Here, the transmission that above-mentioned data compression end can be with asynchronous execution to the compressed data Operation and to the null clear operation of the count value in above-mentioned current target storage domain.
As an example, the storage region included by above-mentioned storage region sequence is followed successively by storage region A, B, C, D, work as transmission When arriving at the time of compressed data, it is assumed that the currently indicated target storage domain of above-mentioned storage region parameter is storage The current target that is stored with respectively takes range " 0ms~10ms ", the count value of " 11ms~20ms " in region D, storage region D 700,530 and target result of calculation " success " count value 1230, above-mentioned data compression end can determine above-mentioned storage region sequence There is no the storage region that sequence is inferior to storage region D in row, above-mentioned data compression end can refer to above-mentioned storage region parameter To the storage region A positioned at above-mentioned storage region sequence header.Above-mentioned data compression end can read each of storage region D Count value, by count value 700,530,1230 and target corresponding with the count value take range " 0ms~10ms ", " 11ms~20ms " and target result of calculation " success " are sent to above-mentioned the data storage end as compressed data.
In step 205, the data storage end stores the compressed data received.
In the present embodiment, above-mentioned the data storage end receives the compressed data that above-mentioned data compression end is sent Afterwards, above-mentioned the data storage end can store the compressed data, such as carry out persistence (Persistence) and deposit Storage.Wherein, persistence be often referred to that a data (object in such as memory) are saved in can (such as magnetic in the storage device of persistence Disk).Here, above-mentioned the data storage end can store the compressed data received to long-range with above-mentioned the data storage end The server of communication connection.
It is a schematic diagram of application scenarios corresponding with embodiment shown in Fig. 2 with continued reference to Fig. 3, Fig. 3.Fig. 3's In application scenarios, data collection terminal can acquire the monitoring data under preset monitoring parameter P1, P2 from monitored server, such as mark Shown in numbers 301.As shown in label 302, above-mentioned data collection terminal can carry out collected monitoring data data encapsulation and generate Shared drive is written in first data A by the first data A.As shown in label 303, data compression end can be from above-mentioned shared drive In extract the first data A, respectively under the preset monitoring parameter P1 included by the first data A data and preset monitoring parameter Data under P2 are compressed to realize that the compression to the first data A can obtain compressed after the completion of compression here Data B.As shown in label 304, in response to arriving at the time of currently transmitted compressed data, above-mentioned data compression end can be with Compressed data B is sent to the data storage end.As shown in label 305, above-mentioned the data storage end can will after compression Data B store to the data storage server connected.
The monitoring system that above-described embodiment of the application provides, can efficiently use preset monitoring parameter, for different Preset monitoring parameter may be used different data compression algorithms and be compressed to the first data extracted, and then can be real Now it is imbued with targetedly data storage.
With further reference to Fig. 4, it illustrates the flows 400 according to one embodiment of the data processing method of the application. The data processing method, includes the following steps:
Step 401, the first data are extracted from preset memory locations.
In the present embodiment, electronic equipment (such as the data compression end shown in FIG. 1 of data processing method operation thereon 105) sequencing for the time that can be written into according to the first data, successively from the first number of above-mentioned preset memory locations extraction According to.It should be pointed out that above-mentioned data compression end can in real time or periodically (such as every 10 seconds etc.) are deposited from above-mentioned preset Storage space sets the first data of extraction.Here, above-mentioned first data can be data collection terminal to from monitored server (such as Fig. 1 Shown in monitored server 101) acquisition default monitoring parameter under monitoring data carry out the number that is generated after data encapsulation According to.
Step 402, the first data extracted are compressed based on default monitoring parameter.
In the present embodiment, above-mentioned electronic equipment from above-mentioned preset memory locations after extracting the first data, above-mentioned electricity Sub- equipment can compress first data based on above-mentioned default monitoring parameter.Wherein, above-mentioned preset monitoring parameter can be with Such as including request amount, the request amount can refer to request that above-mentioned monitored server receives within a predetermined period of time (such as Data inquiry request, data storage request etc.) quantity.Optionally, above-mentioned preset monitoring parameter can also include it is following at least One:Take value, request results.Wherein, this, which takes value, can refer to above-mentioned monitored server at the request that receives Reason spent time, the request results may include successfully and failing.It should be noted that for the first number to extracting , can be referring to the associated description in above-mentioned steps 203 according to the method compressed, details are not described herein.
Step 403, compressed data are periodically sent to the data storage end, so that the data storage end is to receiving To compressed data stored.
In the present embodiment, compressed data periodically can be sent to data storage by above-mentioned electronic equipment End, so that the data storage end stores the compressed data received.For to it is above-mentioned periodically will be compressed The explanation that data afterwards are sent to the data storage end can be referring to the associated description in above-mentioned steps 204, for above-mentioned number The explanation stored to the compressed data received according to storage end can be retouched referring to the correlation in above-mentioned steps 205 It states, details are not described herein.
Data processing method provided by the embodiments of the present application, by extracting the first data from preset memory locations, so as to base The first data extracted are compressed in default monitoring parameter, and compressed data are periodically sent to data Storage end, so that the data storage end stores the compressed data received.It is preset so as to efficiently use Different data compression algorithms may be used to the first data for extracting for different preset monitoring parameters in monitoring parameter It is compressed, and then may be implemented to be imbued with targetedly data storage.
With further reference to Fig. 5, as the realization to method shown in above-mentioned each figure, this application provides a kind of data processing dresses The one embodiment set, device embodiment embodiment of the method as shown in fig. 4 is corresponding, which specifically can be applied to respectively In kind electronic equipment.
As shown in figure 5, the data processing equipment 500 of the present embodiment includes:Data extracting unit 501,502 and of compression unit Transmission unit 503.Wherein, data extracting unit 501 is configured to extract the first data from preset memory locations, wherein above-mentioned First data are data collection terminals carries out data envelope to the monitoring data under the default monitoring parameter that is acquired from monitored server The data generated after dress;Compression unit 502 be configured to based on above-mentioned default monitoring parameter to the first data for extracting into Row compression;Transmission unit 503 is configured to that compressed data are periodically sent to the data storage end, so that above-mentioned number The compressed data received are stored according to storage end.
In the present embodiment, in data processing equipment 500:Data extracting unit 501, compression unit 502 and transmission unit 503 specific processing and its caused technique effect can respectively with reference in 4 corresponding embodiment of figure step 401, step 402 and The related description of step 403, details are not described herein.
In some optional realization methods of the present embodiment, above-mentioned preset monitoring parameter may include following at least one ?:Take value, request results, wherein above-mentioned time-consuming value refers to above-mentioned monitored server and carries out processing institute to the request received The time of consuming, above-mentioned request results include successfully and failing.
In some optional realization methods of the present embodiment, above-mentioned compression unit 502 can be further configured to:If Above-mentioned preset monitoring parameter includes taking value, then takes value for each included by the first data for extracting, wrapped The time-consuming range included, which is provided with, determines that this takes the target at value place and takes range in the preset time-consuming range set of count value, increase In addition stating the count value that target takes range.
In some optional realization methods of the present embodiment, above-mentioned compression unit 502 can be further configured to:If Above-mentioned preset monitoring parameter includes request results, then for each request results included by the first data for extracting, Included request results, which are provided in the request results set of count value, determines destination request knot identical with the request results Fruit increases the count value of above-mentioned destination request result.
In some optional realization methods of the present embodiment, it is previously provided with storage region sequence and is used to indicate current The storage region parameter in used target storage domain;And above-mentioned compression unit 502 can be further configured to:Upper It states and records the count value that currently determined target takes range in the currently indicated target storage domain of storage region parameter And/or the count value of identified destination request result.
In some optional realization methods of the present embodiment, above-mentioned transmission unit 503 can be further configured to:When When arriving at the time of sending compressed data, determines in above-mentioned storage region sequence and be inferior to current mesh with the presence or absence of sequence Mark the storage region of storage region;It is stored inferior to current target if so, above-mentioned storage region parameter is made to be directed toward said sequence The storage region in region;By each count value and target corresponding with count value consumption in above-mentioned current target storage domain When range or destination request result be sent to above-mentioned the data storage end as compressed data, and empty above-mentioned current mesh Mark the count value in storage region.
In some optional realization methods of the present embodiment, above-mentioned transmission unit 503 can be further configured to:It rings Should then it make above-mentioned in determining in above-mentioned storage region sequence there is no the storage region that sequence is inferior to current target storage domain Storage region parameter is directed toward the storage region positioned at above-mentioned storage region sequence header.
Data processing equipment provided by the embodiments of the present application, by extracting the first data from preset memory locations, so as to base The first data extracted are compressed in default monitoring parameter, and compressed data are periodically sent to data Storage end, so that the data storage end stores the compressed data received.It is preset so as to efficiently use Different data compression algorithms may be used to the first data for extracting for different preset monitoring parameters in monitoring parameter It is compressed, and then may be implemented to be imbued with targetedly data storage.
Below with reference to Fig. 6, it illustrates the computer systems 600 suitable for the electronic equipment for realizing the embodiment of the present application Structural schematic diagram.Electronic equipment shown in Fig. 6 is only an example, to the function of the embodiment of the present application and should not use model Shroud carrys out any restrictions.
As shown in fig. 6, computer system 600 includes central processing unit (CPU) 601, it can be read-only according to being stored in Program in memory (ROM) 602 or be loaded into the program in random access storage device (RAM) 603 from storage section 608 and Execute various actions appropriate and processing.In RAM 603, also it is stored with system 600 and operates required various programs and data. CPU 601, ROM 602 and RAM 603 are connected with each other by bus 604.Input/output (I/O) interface 605 is also connected to always Line 604.
It is connected to I/O interfaces 605 with lower component:Importation 606 including keyboard, mouse etc.;It is penetrated including such as cathode The output par, c 607 of spool (CRT), liquid crystal display (LCD) etc. and loud speaker etc.;Storage section 608 including hard disk etc.; And the communications portion 609 of the network interface card including LAN card, modem etc..Communications portion 609 via such as because The network of spy's net executes communication process.Driver 610 is also according to needing to be connected to I/O interfaces 605.Detachable media 611, such as Disk, CD, magneto-optic disk, semiconductor memory etc. are mounted on driver 610, as needed in order to be read from thereon Computer program be mounted into storage section 608 as needed.
Particularly, in accordance with an embodiment of the present disclosure, it may be implemented as computer above with reference to the process of flow chart description Software program.For example, embodiment of the disclosure includes a kind of computer program product comprising be carried on computer-readable medium On computer program, which includes the program code for method shown in execution flow chart.In such reality It applies in example, which can be downloaded and installed by communications portion 609 from network, and/or from detachable media 611 are mounted.When the computer program is executed by central processing unit (CPU) 601, executes and limited in the system of the application Above-mentioned function.
It should be noted that computer-readable medium shown in the application can be computer-readable signal media or meter Calculation machine readable storage medium storing program for executing either the two arbitrarily combines.Computer readable storage medium for example can be --- but not Be limited to --- electricity, magnetic, optical, electromagnetic, infrared ray or semiconductor system, device or device, or arbitrary above combination.Meter The more specific example of calculation machine readable storage medium storing program for executing can include but is not limited to:Electrical connection with one or more conducting wires, just It takes formula computer disk, hard disk, random access storage device (RAM), read-only memory (ROM), erasable type and may be programmed read-only storage Device (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), light storage device, magnetic memory device, Or above-mentioned any appropriate combination.In this application, can be any include computer readable storage medium or storage journey The tangible medium of sequence, the program can be commanded the either device use or in connection of execution system, device.And at this In application, computer-readable signal media may include in a base band or as the data-signal that a carrier wave part is propagated, Wherein carry computer-readable program code.Diversified forms may be used in the data-signal of this propagation, including but unlimited In electromagnetic signal, optical signal or above-mentioned any appropriate combination.Computer-readable signal media can also be that computer can Any computer-readable medium other than storage medium is read, which can send, propagates or transmit and be used for By instruction execution system, device either device use or program in connection.Include on computer-readable medium Program code can transmit with any suitable medium, including but not limited to:Wirelessly, electric wire, optical cable, RF etc. or above-mentioned Any appropriate combination.
Flow chart in attached drawing and block diagram, it is illustrated that according to the system of the various embodiments of the application, method and computer journey The architecture, function and operation in the cards of sequence product.In this regard, each box in flowchart or block diagram can generation A part for a part for one module, program segment, or code of table, above-mentioned module, program segment, or code includes one or more Executable instruction for implementing the specified logical function.It should also be noted that in some implementations as replacements, institute in box The function of mark can also occur in a different order than that indicated in the drawings.For example, two boxes succeedingly indicated are practical On can be basically executed in parallel, they can also be executed in the opposite order sometimes, this is depended on the functions involved.Also it wants It is noted that the combination of each box in block diagram or flow chart and the box in block diagram or flow chart, can use and execute rule The dedicated hardware based systems of fixed functions or operations is realized, or can use the group of specialized hardware and computer instruction It closes to realize.
Being described in unit involved in the embodiment of the present application can be realized by way of software, can also be by hard The mode of part is realized.Described unit can also be arranged in the processor, for example, can be described as:A kind of processor packet Include data extracting unit, compression unit and transmission unit.Wherein, the title of these units is not constituted to this under certain conditions The restriction of unit itself, for example, data extracting unit is also described as " extracting the list of the first data from preset memory locations Member ".
As on the other hand, present invention also provides a kind of computer-readable medium, which can be Included in electronic equipment described in above-described embodiment;Can also be individualism, and without be incorporated the electronic equipment in. Above computer readable medium carries one or more program, when the electronics is set by one for said one or multiple programs It is standby when executing so that the electronic equipment includes:Instruction is generated in response to receiving sequence number, then executes following processing step:From Preset memory locations extract the first data, wherein above-mentioned first data are data collection terminals to being acquired from monitored server Monitoring data under default monitoring parameter carries out the data generated after data encapsulation;Based on above-mentioned default monitoring parameter to extraction The first data gone out are compressed;Compressed data are periodically sent to the data storage end, so that above-mentioned data are deposited Chu Duan stores the compressed data received.
Above description is only the preferred embodiment of the application and the explanation to institute's application technology principle.People in the art Member should be appreciated that invention scope involved in the application, however it is not limited to technology made of the specific combination of above-mentioned technical characteristic Scheme, while should also cover in the case where not departing from foregoing invention design, it is carried out by above-mentioned technical characteristic or its equivalent feature Other technical solutions of arbitrary combination and formation.Such as features described above has similar work(with (but not limited to) disclosed herein Can technical characteristic replaced mutually and the technical solution that is formed.

Claims (11)

1. a kind of monitoring system, which is characterized in that the monitoring system includes:Data collection terminal, data compression end and data are deposited Chu Duan;
The data collection terminal, for acquiring the monitoring data under preset monitoring parameter from monitored server, to the monitoring Data carry out data encapsulation and generate the first data, and preset memory locations are written in first data;
The data compression end is based on the preset monitoring parameter pair for extracting the first data from the preset memory locations The first data extracted are compressed, and compressed data are periodically sent to the data storage end;
The data storage end, for being stored to the compressed data received.
2. system according to claim 1, which is characterized in that the preset monitoring parameter includes at least one of following:Consumption Duration, request results, wherein the time-consuming value refer to the monitored server request received handle it is spent Time, the request results include successfully and failing.
3. system according to claim 2, which is characterized in that it is described based on the preset monitoring parameter to extract One data are compressed, including:
If the preset monitoring parameter includes taking value, value is taken for each included by the first data for extracting, Determine that the target where this takes value takes in the preset time-consuming range set that included time-consuming range is provided with count value Range increases the count value that the target takes range.
4. system according to claim 3, which is characterized in that it is described based on the preset monitoring parameter to extract One data are compressed, including:
If the preset monitoring parameter includes request results, each request included by the first data for extracting is tied Fruit determines that target identical with the request results is asked in the request results set that included request results are provided with count value It asks as a result, increasing the count value of the destination request result.
5. system according to claim 4, which is characterized in that be previously provided with storage region sequence and be used to indicate current The storage region parameter in used target storage domain;And
It is described that the first data extracted are compressed based on the preset monitoring parameter, including:
Currently determined target is recorded in the currently indicated target storage domain of the storage region parameter takes range Count value and/or identified destination request result count value.
6. system according to claim 5, which is characterized in that described that compressed data are periodically sent to institute The data storage end is stated, including:
When arriving when send compressed data at the time of, determine in the storage region sequence with the presence or absence of sequence inferior to working as The storage region in preceding target storage domain;
If so, the storage region parameter is made to be directed toward the storage region that the sequence is inferior to current target storage domain;
By in the current target storage domain each count value and target corresponding with the count value take range or mesh Mark request results are sent to the data storage end as compressed data, and empty the current target storage domain In count value.
7. system according to claim 6, which is characterized in that described that compressed data are periodically sent to institute The data storage end is stated, including:
In response to there is no the storage regions that sequence is inferior to current target storage domain in the determination storage region sequence, then The storage region parameter is set to be directed toward the storage region positioned at the storage region sequence header.
8. a kind of data processing method, which is characterized in that the method includes:
The first data are extracted from preset memory locations, wherein first data are data collection terminals to from monitored server Monitoring data under the default monitoring parameter of acquisition carries out the data generated after data encapsulation;
The first data extracted are compressed based on the default monitoring parameter;
Compressed data are periodically sent to the data storage end, so that the data storage end is to receiving through pressure Data after contracting are stored.
9. a kind of data processing equipment, which is characterized in that described device includes:
Data extracting unit is configured to extract the first data from preset memory locations, wherein first data are that data are adopted The data that collection end to the monitoring data under the default monitoring parameter that acquires from monitored server generated after data encapsulation;
Compression unit is configured to compress the first data extracted based on the default monitoring parameter;
Transmission unit is configured to that compressed data are periodically sent to the data storage end, so that the data are deposited Chu Duan stores the compressed data received.
10. a kind of electronic equipment, which is characterized in that including:
One or more processors;
Storage device, for storing one or more programs,
When one or more of programs are executed by one or more of processors so that one or more of processors are real Now method as claimed in claim 8.
11. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that described program is handled Device realizes method as claimed in claim 8 when executing.
CN201710269398.2A 2017-04-24 2017-04-24 monitoring system, data processing method and device Pending CN108733534A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201710269398.2A CN108733534A (en) 2017-04-24 2017-04-24 monitoring system, data processing method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201710269398.2A CN108733534A (en) 2017-04-24 2017-04-24 monitoring system, data processing method and device

Publications (1)

Publication Number Publication Date
CN108733534A true CN108733534A (en) 2018-11-02

Family

ID=63934180

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201710269398.2A Pending CN108733534A (en) 2017-04-24 2017-04-24 monitoring system, data processing method and device

Country Status (1)

Country Link
CN (1) CN108733534A (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101155066A (en) * 2006-09-29 2008-04-02 中国移动通信集团北京有限公司 Network performance analysis system and method
CN102043679A (en) * 2010-12-22 2011-05-04 北京中电普华信息技术有限公司 Method and system for acquiring performance analysis data of application system
CN104820630A (en) * 2015-05-22 2015-08-05 上海新炬网络信息技术有限公司 System resource monitoring device based on business variable quantity
US20150254157A1 (en) * 2014-03-07 2015-09-10 Texas Instruments Incorporated Method for Throttling Trace Data Streams
CN105930491A (en) * 2016-04-28 2016-09-07 安徽四创电子股份有限公司 Monitoring data storage method based on time sequence database InfluxDB

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101155066A (en) * 2006-09-29 2008-04-02 中国移动通信集团北京有限公司 Network performance analysis system and method
CN102043679A (en) * 2010-12-22 2011-05-04 北京中电普华信息技术有限公司 Method and system for acquiring performance analysis data of application system
US20150254157A1 (en) * 2014-03-07 2015-09-10 Texas Instruments Incorporated Method for Throttling Trace Data Streams
CN104820630A (en) * 2015-05-22 2015-08-05 上海新炬网络信息技术有限公司 System resource monitoring device based on business variable quantity
CN105930491A (en) * 2016-04-28 2016-09-07 安徽四创电子股份有限公司 Monitoring data storage method based on time sequence database InfluxDB

Similar Documents

Publication Publication Date Title
CN107729139A (en) A kind of method and apparatus for concurrently obtaining resource
CN106445817B (en) Processing method, the device and system of test data
CN110401720A (en) Information processing method, device, system, application server and medium
CN108900388A (en) Method and apparatus for monitor network quality
CN106548261A (en) A kind of Business Process Control method, device and electronic equipment
CN109885624A (en) Data processing method, device, computer equipment and storage medium
CN110149298A (en) A kind of method and apparatus for kidnapping detection
CN108108248A (en) A kind of CPU+GPU cluster management methods, device and equipment for realizing target detection
CN108833510A (en) Message treatment method and device
CN106992940B (en) Message processing method, device, system and terminal
CN107729570A (en) Data migration method and device for server
US10775751B2 (en) Automatic generation of regular expression based on log line data
CN113608751B (en) Operation method, device and equipment of reasoning service platform and storage medium
CN114490280A (en) Log processing method, device, equipment and medium
CN110719233B (en) Method and device for transmitting information
CN108021462A (en) Method and apparatus for calling cloud service
CN108733534A (en) monitoring system, data processing method and device
CN109218338A (en) Information processing system, method and apparatus
CN112102095A (en) Fund product recommendation method, device and equipment
CN107315672A (en) Method and apparatus for monitoring server
CN110009273A (en) Information processing method and device, storage medium, electronic device
CN110389891A (en) The method and apparatus of test application program
CN108880941A (en) A kind of data transmitting test device and its data processing method of block chain
EP2590366A1 (en) Method and system for monitoring message objects sent from a client to invoke operations on a server in a distributed computing environment
CN111273988B (en) Account display method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20181102