CN107204875B - Data reporting link monitoring method and device, electronic equipment and storage medium - Google Patents
Data reporting link monitoring method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN107204875B CN107204875B CN201710329500.3A CN201710329500A CN107204875B CN 107204875 B CN107204875 B CN 107204875B CN 201710329500 A CN201710329500 A CN 201710329500A CN 107204875 B CN107204875 B CN 107204875B
- Authority
- CN
- China
- Prior art keywords
- server
- request
- amount
- shielding
- preset
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/14—Network analysis or design
- H04L41/142—Network analysis or design using statistical or mathematical methods
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/06—Generation of reports
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0805—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability
- H04L43/0811—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability by checking connectivity
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0805—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability
- H04L43/0817—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability by checking functioning
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
Abstract
The disclosure relates to a method and a device for monitoring a data reporting link. The method comprises the following steps: when the request message is sent, selecting a first server from a first server set according to a preset selection rule, wherein the first server set is a complementary set of a shielding server set in a total server set; requesting to establish a communication link with a first server, and counting the request failure amount of the first server; when the request failure amount exceeds a preset value, adding a first server into a shielding server set; and clearing the request failure amount when the request for establishing the communication link is successful. The method and the system can automatically shield the server in the abnormal state, automatically remove the shielding state of the server when the server is recovered to be normal, and improve the data sending performance and efficiency of the production end.
Description
Technical Field
The present disclosure relates to the field of big data, and in particular, to a data reporting link monitoring method, a data reporting link monitoring apparatus, an electronic device, and a computer-readable storage medium.
Background
With the development of the internet, the amount of data to be processed increases in a geometric progression, and how to efficiently process massive data becomes a focus of attention in the technical field of big data.
An existing mass data processing technical scheme adopts a reporting model that data is reported to a storage server Broker by a Producer, namely a real-time data production client: as shown in fig. 1, the production end Producer is responsible for reporting data to the storage server, the production end Producer periodically obtains a storage server list corresponding to different data from the configuration system, when data is sent, the production end Producer checks a mapping relation table between the data and the storage server, selects a server from a server set having a mapping relation with the data according to a certain algorithm, such as a round robin algorithm, to send the data, the production end Producer checks whether a link to the data storage server exists, if not, a data link is established, and then actual data sending processing is performed.
However, the data reporting model has the following problems: on one hand, when the server fails, the production end cannot timely know that the server fails, and the data sending efficiency and performance of the production end are affected due to frequent retry and establishment of a data link with the server; on the other hand, the server may have a reduced data processing capability during operation, such as an overload of the machine, a failure of the hard disk, and other abnormal conditions, which may reduce the data transmission efficiency and performance of the production end communicating with the server.
It is noted that the information disclosed in the above background section is only for enhancement of understanding of the background of the present disclosure and therefore may include information that does not constitute prior art that is already known to a person of ordinary skill in the art.
Disclosure of Invention
An object of the present disclosure is to provide a method and a device for monitoring a data reporting link, so as to overcome one or more problems caused by the limitations and disadvantages of the related art at least to a certain extent.
According to an aspect of the present disclosure, a method for monitoring a data reporting link is provided, which includes:
selecting a first server from a first server set according to a preset selection rule when a request message is sent, wherein the first server set is a complementary set of a shielding server set in a total server set;
requesting to establish a communication link with the first server, and counting the request failure amount of the first server;
when the request failure amount exceeds a preset value, adding the first server into the shielding server set; and
and clearing the request failure amount when the request for establishing the communication link is successful.
According to an aspect of the present disclosure, there is provided a data reporting link monitoring apparatus, including:
the device comprises a selecting unit, a judging unit and a sending unit, wherein the selecting unit is used for selecting a first server from a first server set according to a preset selection rule when a request message is sent, and the first server set is a complement of a shielding server set in a total server set;
the statistical unit is used for requesting to establish a communication link with the first server and counting the request failure amount of the first server;
the link failure processing unit is used for adding the first server into the shielding server set when the request failure amount exceeds a preset value; and
and the link success processing unit is used for clearing the request failure amount when the request for establishing the communication link is successful.
According to an aspect of the present disclosure, there is provided an electronic device including:
a processor; and
and a memory, wherein computer readable instructions are stored on the memory, and when executed by the processor, the computer readable instructions implement the data reporting link monitoring method.
According to an aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the above-mentioned data reporting link monitoring method.
According to the method and the device for monitoring the data reporting link in the embodiment of the example, the server is selected from the server set in the normal state according to the preset selection rule, the server with the request failure amount exceeding the preset value is added into the shielding server set, and the request failure amount is cleared when the request for establishing the communication link is successful. On one hand, the server is selected from the server set in the normal state according to the preset selection rule, and compared with the prior art, the condition of reporting data to the server in the abnormal state can be remarkably reduced, so that the data sending efficiency and performance of a production end are improved; on the other hand, the server with the request failure amount exceeding the preset value is added into the shielding server set, so that the server in an abnormal state can be automatically shielded; on the other hand, the method clears the request failure amount when the request for establishing the communication link is successful, can automatically remove the shielding state of the server when the server returns to normal, and has simple processing and high processing efficiency.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The above and other features and advantages of the present disclosure will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings.
Fig. 1 schematically shows a schematic block diagram of a data reporting method in a technical solution;
fig. 2 schematically illustrates a flow chart of a method of data reporting link monitoring according to an exemplary embodiment of the present disclosure;
fig. 3 schematically illustrates an interaction diagram of a production end and a server of a data reporting link monitoring method according to an exemplary embodiment of the present disclosure;
fig. 4 schematically illustrates an interaction diagram of a production end and a server of a data reporting link monitoring method according to another exemplary embodiment of the present disclosure;
fig. 5 schematically illustrates a block diagram of a data reporting link monitoring device according to an exemplary embodiment of the present disclosure;
FIG. 6 schematically illustrates a block diagram of an electronic device according to an exemplary embodiment of the present disclosure; and
fig. 7 schematically illustrates a schematic diagram of a computer-readable storage medium according to an exemplary embodiment of the present disclosure.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The same reference numerals denote the same or similar parts in the drawings, and thus, a repetitive description thereof will be omitted.
Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the embodiments of the disclosure can be practiced without one or more of the specific details, or with other methods, components, materials, devices, steps, and the like. In other instances, well-known structures, methods, devices, implementations, materials, or operations are not shown or described in detail to avoid obscuring aspects of the disclosure.
The block diagrams shown in the figures are functional entities only and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in the form of software, or in one or more software-hardened modules, or in different networks and/or processor devices and/or microcontroller devices.
In a technical scheme of a reporting model that data is reported to a storage server Broker by a Producer, in order to maintain data sending efficiency and performance of the Producer, the Producer sets a message buffer queue for each link locally, buffers received data packets to be sent, and sends messages in batch when the links are available so as to improve data sending performance and performance of a generator; meanwhile, the link is maintained for a certain time after the data is sent, so that the performance loss caused by repeated link establishment is avoided, and the performance is improved by setting the size of the buffer area to avoid the memory backlog. However, the situation that the production end cannot know that the server is abnormal also exists in the technical scheme.
In order to solve the problem that a production end cannot know the abnormal condition of a server, one technical scheme is to shield the server with abnormal link, and the other technical scheme is to judge whether the server has the abnormal condition according to the processing speed of the server to the request response. However, in actual use, on the premise of mass reported data, the technical solutions have problems of too limited system performance, incapability of automatically removing shielding, too high abnormal detection cost and the like, so that the system performance cannot be well exerted, and the problems of performance and efficiency reduction cannot be quickly found and solved when the server is abnormal.
Based on the above, in this example, a method for monitoring a data reporting link is first provided, which may be applied to a Producer in a data reporting system. Referring to fig. 2, the method for monitoring a data reporting link may include the following steps:
s210, selecting a first server from a first server set according to a preset selection rule when a request message is sent, wherein the first server set is a complement of a shielding server set in a total server set;
s220, requesting to establish a communication link with the first server, and counting the request failure amount of the first server;
step S230, when the request failure amount exceeds a preset value, adding the first server into the shielding server set; and
and S240, clearing the request failure amount when the request for establishing the communication link is successful.
According to the data reporting link monitoring method in the embodiment, on one hand, the server is selected from the server set in the normal state according to the preset selection rule, and compared with the prior art, the situation that data is reported to the server in the abnormal state can be remarkably reduced, so that the data sending efficiency and performance of a production end are improved; on the other hand, the server with the request failure amount exceeding the preset value is added into the shielding server set, so that the server in an abnormal state can be automatically shielded; on the other hand, the method clears the request failure amount when the request for establishing the communication link is successful, can automatically remove the shielding state of the server when the server returns to normal, and has simple processing and high processing efficiency.
Next, a data reporting link monitoring method in this exemplary embodiment will be further described.
In step S210, a first server is selected from the first server set according to a predetermined selection rule when the request message is sent, where the first server set is a complement of the masked server set in the total server set.
In this exemplary embodiment, the total server set may be a set of all server Broker of the Tube system of the distributed message processing system. The total server set may include a set of servers in a masked state and a set of servers in a normal state, where the set of servers in the normal state is a complement of the masked server set in the total server set, and thus, the first server set may be a set of servers in the normal state.
Referring to fig. 3, when data reporting is required, the data Producer first sends a request message to the server Broker, step S1. When the request message is sent, in order to improve the data sending performance of the Producer, the request message may be sent only to the server whose message processing capability is in a normal state. Therefore, in the present exemplary embodiment, a server with poor message processing capability or a server in an abnormal state, i.e., being overloaded or failing, may be listed as the set of mask servers. Then, when the request message is sent, a server may be selected from a set excluding the masked server set in the masked state from the total server set, that is, a server may be selected from a complement of the masked server set in the total server set. And if the complementary set of the shielding server set in the total server set is empty, refusing to send the request message, otherwise, selecting the server from the complementary set of the shielding server set in the total server set according to a preset selection rule.
In the present exemplary embodiment, the predetermined selection rule may be sequential selection, binary selection, index selection, and the like, but the predetermined selection rule in the exemplary embodiment of the present disclosure is not limited thereto, and for example, the predetermined selection rule may also be hash table selection, tree table selection, and the like, which is also within the protection scope of the present disclosure.
Further, in this exemplary embodiment, in order to further improve the data transmission performance and efficiency of the production side, when the request message is sent, referring to fig. 3, in step S1, it may be determined whether the total response message waiting amount totalRspWaitCont for each server in the total server set exceeds a preset number, and when it is determined that the total response message waiting amount exceeds the preset number, the production side may refuse to send the request message, so as to avoid that the production side sends too many invalid request messages when the plurality of servers are abnormal. And when the total response message waiting amount is judged not to exceed the preset amount, selecting a server from a complementary set of the shielding server set in the total server set according to the preset selection rule. In this example embodiment, the total response message waiting amount totalRspWaitCont may be the total amount of all outgoing requests for which no response is received by the production side.
Furthermore, in this exemplary embodiment, in order to better monitor the communication link between the production end and the server, referring to fig. 3, in step S1, selecting the first server from the first set of servers according to the predetermined selection rule may further include: and counting the total response message waiting amount, the response message waiting amount itemRspWaitCnt of the first server and the request sending amount itemReqSentCnt of the first server, wherein the response message waiting amount itemrwaitcnt of the server may be the total request amount of the response which is sent out to a certain server by the production end but has not received the response returned by the server. The request sending amount itemReqSentCnt of the server may be a request amount sent out from the production end to a certain server within a predetermined time.
In the present exemplary embodiment, the predetermined time may be a unit time of, for example, 1s, or may also be 2s or other suitable time periods determined according to the processing performance of the server and the production end, which is not particularly limited in this disclosure.
Further, in this exemplary embodiment, the counting the total response message waiting amount, the response message waiting amount of the first server, and the request sending amount of the first server may include: the total response message waiting amount, the response message waiting amount of the first server, and the request transmission amount of the first server are increased, for example, the total response message waiting amount, the response message waiting amount of the first server, and the request transmission amount of the first server are each counted up by 1.
In addition, in order to monitor whether the server is in an abnormal state, after the selecting the first server from the first set of servers, the method may further include: judging whether the request response detection time length itemCheckReqRspDur of the first server to the request message needs to be counted or not; and recording the sending time of the request message when the request response detection time is judged to need to be counted. In this exemplary embodiment, the request response detection duration itemceckreqrspdur of the server may be a time consumption from the time when the production end sends the detection request to a certain server to the time when the successful response message is received.
It should be noted that, in this example embodiment, the Producer may be in a standalone mode or in a cluster mode, which is not particularly limited in this disclosure, and the server Broker may be deployed in the cluster mode.
Next, in step S220, a request for establishing a communication link with the first server is made, and a request failure amount of the first server is counted.
In the present exemplary embodiment, referring to fig. 3, after selecting the server to perform data transmission, a communication link may be requested to be established with the first server, step S2. In order to simply and efficiently monitor whether the server is in an abnormal state, when the communication link is established in step S2, statistics may be performed on the request failure amount itemconmail, and the request failure amount itemconmail of the server may be the number of times that the production end requests a certain server to establish the communication link failure within a predetermined time.
Further, counting the amount of request failures may include: the request failure amount is initialized to 0, and when a communication link is established with the first server, if the establishment of the communication link fails, the request failure amount is accumulated, for example, the count 1 is accumulated, and the establishment of the communication link with the first server is requested again.
Next, in step S230, when the request failure amount exceeds a preset value, the first server is added to the masking server set.
In the present exemplary embodiment, the preset value is a value set according to the processing performance of the production side and the server, and may be changed as the processing performance of the generation side and/or the server changes. In addition, the preset value can be obtained statistically according to the processing result after the data reporting link monitoring method is applied, which is also within the protection scope of the present disclosure.
In an example embodiment, the first server may be added to the masking server set when the number of failed requests exceeds a preset value. The server added into the shielding server set is in the shielding state, and the server in the shielding server set cannot be selected when the subsequent production end requests to establish a communication link.
Next, in step S240, the request failure amount is cleared when the request for establishing the communication link is successful.
In the present exemplary embodiment, the request failure amount of the first server is cleared when the request to establish the communication link with the first server is successful. In addition, in this exemplary embodiment, a request message may be sent to each server in the set of the shield servers at a certain frequency at a certain timing, a communication link is requested to be established with each server in the set of the shield servers, the request failure amount of the server that successfully establishes the communication link is cleared, and after the request failure amount of the server that successfully establishes the communication link is cleared, the shield state of the server may be released, that is, the server may be taken out of the set of the shield servers. Therefore, the exemplary embodiment can automatically unmask the server in the masked state according to the request failure amount of the server, and resume the data reporting process on the communication link with the server.
Further, in the present exemplary embodiment, in order to better monitor the communication link between the production end and the server, referring to fig. 3, in step S3, when a response message to the request for establishing the communication link message returned from the first server is received, the total response message waiting amount and the response message waiting amount of the first server may be reduced, for example, the total response message waiting amount and the response message waiting amount of the first server are reduced by 1. Moreover, when the response message is a successful response message, the successful response amount itemSuccRspCnt of the first server may be increased, for example, by adding 1 to the successful response amount of the first server. In this example embodiment, the successful response amount itemSuccRspCnt of the server may be an outgoing request successful response amount received by the production end from a certain server within a predetermined time.
Further, in this example embodiment, in order to monitor whether the server is in an abnormal state through the processing duration of the request message by the server, the method for monitoring a data reporting link may further include: when a response message returned from the first server is received, the request response detection duration of the first server is calculated for the request needing to calculate the request response detection duration, and the value of the calculation result is assigned to the request response detection duration itemCheckReqRspDur of the first server. In the present exemplary embodiment, the request-response detection duration may be calculated according to the transmission time of the request message and the time of receiving the response message corresponding to the request message returned from the server, for example, the request-response detection duration may be a difference between the transmission time of the request message and the time of receiving the response message corresponding to the request message.
Specifically, in the present exemplary embodiment, in order to monitor whether the server is in an abnormal state through the processing duration of the request message by the server, referring to fig. 4, a new thread may be started to periodically detect the quality of the communication link established between the production end and the server, for example, the quality of the communication link may be detectedAcquiring request response detection time of each server within preset time; and then adding the server with the request response detection duration being out of the preset range and the average value of the request response detection durations being maximum into the shielding server set according to normal distribution. For example, let the normal distribution be N (μ, σ) 2 ) Mu is the mean value of the request response detection time length of the server obtained by statistics, sigma 2 The predetermined range of the request response detection time duration may be (μ - σ, μ + σ), (μ -2 σ, μ +2 σ), or (μ -3 σ, μ +3 σ), or may be other suitable range, which is not particularly limited by the present disclosure. The request response detection time duration may be outside the predetermined range (μ - σ, μ + σ), (μ -2 σ, μ +2 σ), or (μ -3 σ, μ +3 σ), and the server having the largest mean value of the request response detection time durations may be added to the masking server set.
In addition, in this exemplary embodiment, whether the server is in an abnormal state may also be monitored in other manners. For example, it may be monitored whether a server is in an abnormal state by a ratio of a request transmission amount to a successful response amount of each server, and therefore, the method for monitoring a data reporting link may further include: calculating the request sending quantity and the successful response quantity of each server in a preset time; obtaining the ratio of the request sending quantity and the successful response quantity of each server in the preset time; and adding the server with the ratio larger than a preset threshold value into the shielding server set. In the present exemplary embodiment, the predetermined threshold may be a value set according to the processing performance of the production side and the server, and the preset value may be changed as the processing performance of the production side and/or the server is changed.
Further, in this exemplary embodiment, in order to automatically take out a server in the set of masked servers from the set of masked servers after the server in the set of masked servers recovers to normal, as shown in fig. 4, a new thread may be started to periodically detect a server in a masked state, and therefore, the method for monitoring a data reporting link may further include: detecting each server in the shielding server set at regular time; and when a server which successfully establishes the communication link is detected, or a server which requests the response detection time length mean value to accord with the preset range of the normal distribution, or a server of which the ratio of the request sending quantity to the successful response quantity is smaller than the preset threshold value is detected, the shielding state of the server is released. Further, the periodically detecting each server in the set of masked servers may include: and regularly sending a request message to each server in the shielding server set according to a preset frequency, and updating the value of the server abnormal detection quantity itemAbnorm CheckReqCnt. The server anomaly detection quantity itemAbnorm CheckReqCnt can be the message quantity received by a production end Producer sending a detection request to a certain shielded server.
It should be noted that although the various steps of the methods of the present disclosure are depicted in the drawings in a particular order, this does not require or imply that these steps must be performed in this particular order, or that all of the depicted steps must be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions, etc.
In addition, in this example embodiment, a data reporting link monitoring device is further provided. Referring to fig. 5, the data reporting link monitoring apparatus 500 may include: a selecting unit 510, a counting unit 520, a link failure processing unit 530, and a link success processing unit 540. Wherein:
the selecting unit 510 is configured to select a first server from a first server set according to a predetermined selection rule when sending the request message, where the first server set is a complement of the masking server set in the total server set;
the counting unit 520 is configured to request to establish a communication link with the first server, and count a request failure amount of the first server;
the link failure processing unit 530 is configured to add the first server to the masked server set when the request failure amount exceeds a preset value; and
the link success processing unit 540 is configured to clear the request failure amount when the request for establishing the communication link is successful.
Further, in this example embodiment, selecting the first server from the first set of servers according to the predetermined selection rule when sending the request message may include:
judging whether the total response message waiting amount of each server in the total server set exceeds a preset amount when the request message is sent;
refusing to send the request message when the total response message waiting amount is judged to exceed the preset amount;
and when the total response message waiting amount is judged not to exceed the preset amount, selecting the first server from the first server set according to a preset selection rule.
Further, in this exemplary embodiment, after selecting the first server from the first set of servers, the method may further include:
and counting the total response message waiting amount, the response message waiting amount of the first server and the request sending amount of the first server.
Further, in this exemplary embodiment, after selecting the first server from the first set of servers, the method may further include:
judging whether the request response detection duration of the first server to the request message needs to be counted or not;
and recording the sending time of the request message when the request response detection time needs to be counted.
In addition, in this exemplary embodiment, the data reporting link monitoring apparatus 500 may further include:
a first receiving processing unit, configured to reduce the total response message waiting amount and the response message waiting amount of the first server when receiving the response message to the request message returned from the first server.
In addition, in this exemplary embodiment, the data reporting link monitoring apparatus 500 may further include:
and the second receiving and processing unit is used for increasing the successful response quantity of the first server when the response message is a successful response message.
In addition, in this exemplary embodiment, the data reporting link monitoring apparatus 500 may further include:
a third receiving and processing unit, configured to determine whether it is necessary to count a request response detection duration of the request message from the first server when receiving a response message to the request message returned from the first server; and when the request response detection duration needs to be counted, calculating the request response detection duration of the first server according to the sending time of the request message and the time of receiving the response message.
In addition, in this exemplary embodiment, the data reporting link monitoring apparatus 500 may further include:
the computing unit is used for computing the request sending quantity and the successful response quantity of each server in preset time;
a ratio obtaining unit, configured to obtain a ratio of a request sending amount to a successful response amount of each server within the predetermined time;
a first joining unit, configured to join the server with the ratio greater than a predetermined threshold into the masked server set.
In addition, in this example embodiment, the device for monitoring a data reporting link may further include:
a duration obtaining unit, configured to obtain a request response detection duration of each server within the predetermined time;
and the second joining unit is used for joining the server with the request response detection duration being out of the preset range and the average value of the request response detection durations being maximum into the shielding server set according to normal distribution.
In addition, in this example embodiment, the device for monitoring a data reporting link may further include:
the detection unit is used for detecting each server in the shielding server set at regular time;
and the shielding removing unit is used for removing the shielding state of the server when the server which successfully establishes the communication link is detected, or the server which requests the response detection time length mean value to conform to the predetermined range of the normal distribution, or the server which requests the ratio of the sending quantity to the successful response quantity to be less than the predetermined threshold value is detected.
Further, in this example embodiment, the detecting each server in the set of masked servers at regular time may include:
and sending request messages to all servers in the shielding server set at regular time according to a preset frequency, and updating the value of the abnormal detection quantity of the server.
Since each functional module of the data reporting link monitoring apparatus 500 in the exemplary embodiment of the present disclosure corresponds to the step of the exemplary embodiment of the data reporting link monitoring method, it is not described herein again.
It should be noted that although in the above detailed description several modules or units of the data reporting link monitoring device are mentioned, this division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.
In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.
As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or program product. Thus, various aspects of the invention may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system.
An electronic device 600 according to this embodiment of the invention is described below with reference to fig. 6. The electronic device 600 shown in fig. 6 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 6, the electronic device 600 is in the form of a general purpose computing device. The components of the electronic device 600 may include, but are not limited to: the at least one processing unit 610, the at least one memory unit 620, and a bus 630 that couples the various system components including the memory unit 620 and the processing unit 610.
Wherein the storage unit stores program code, which can be executed by the processing unit 610, to cause the processing unit 610 to execute the steps according to various exemplary embodiments of the present invention described in the above-mentioned exemplary server state acquisition methods of the present specification. For example, the processing unit 610 may execute step s210 shown in fig. 2. when sending the request message, a first server is selected from a first set of servers according to a predetermined selection rule, where the first set of servers is a complement of the masked set of servers in the total set of servers; s220, requesting to establish a communication link with the first server, and counting the request failure amount of the first server; step S230, when the request failure amount exceeds a preset value, adding the first server into the shielding server set; and step S240, clearing the request failure amount when the request for establishing the communication link is successful.
The storage unit 620 may include readable media in the form of volatile memory units, such as a random access memory unit (RAM)6201 and/or a cache memory unit 6202, and may further include a read-only memory unit (ROM) 6203.
The memory unit 620 may also include programs/utilities 6204 having a set (at least one) of program modules 6205, such program modules 6205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
The electronic device 600 may also communicate with one or more external devices 670 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 600, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 600 to communicate with one or more other computing devices. Such communication may occur via an input/output (I/O) interface 650. Also, the electronic device 600 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via the network adapter 660. As shown, the network adapter 660 communicates with the other modules of the electronic device 600 over the bus 630. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.
In an exemplary embodiment of the present disclosure, there is also provided a computer readable storage medium having stored thereon a program product capable of implementing the above-described method of the present specification. In some possible embodiments, the various aspects of the present invention may also be implemented in the form of a program product, which includes program code for causing a terminal device to perform the steps according to various exemplary embodiments of the present invention described in "method for monitoring a data reporting link in an exemplary embodiment" described above in this specification, when the program product is run on the terminal device.
Referring to fig. 7, a program product 700 for implementing the above method according to an embodiment of the present invention is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited in this regard and, in the present document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
A computer readable signal medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
Furthermore, the above-described figures are merely schematic illustrations of processes involved in methods according to exemplary embodiments of the invention, and are not intended to be limiting. It will be readily understood that the processes shown in the above figures are not intended to indicate or limit the chronological order of the processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, e.g., in multiple modules.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice in the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims (13)
1. A data reporting link monitoring method is applied to a production end which needs to report data to a server, and comprises the following steps:
when the request message is sent, selecting a first server from a first server set according to a preset selection rule, wherein the first server set is a complementary set of a shielding server set in a total server set;
requesting establishment of a communication link with the first server;
when a communication link is established with a first server, if the establishment of the communication link fails, increasing the request failure amount, and requesting the establishment of the communication link with the first server again, wherein the request failure amount is the number of times of the communication link establishment failure requested by a production end to the first server within a preset time;
when the request failure amount exceeds a preset value, adding the first server into the shielding server set;
acquiring request response detection time of each server within preset time;
adding the server with the request response detection duration being out of the preset range and the average value of the request response detection duration being maximum into the shielding server set according to normal distribution;
clearing the request failure amount of the first server when the request for establishing the communication link is successful;
after the request failure amount of the first server is cleared, taking the first server out of the shielding server set;
and detecting each server in the shielding server set at regular time, and taking the first server out of the shielding server set when the mean value of the request response detection time length of the first server accords with the preset range selected according to the normal distribution.
2. The method of claim 1, wherein selecting the first server from the first set of servers according to the predetermined selection rule when sending the request message comprises:
judging whether the total response message waiting amount of each server in the total server set exceeds a preset amount when the request message is sent;
refusing to send the request message when the total response message waiting amount is judged to exceed the preset amount;
and when the total response message waiting amount is judged not to exceed the preset amount, selecting the first server from the first server set according to a preset selection rule.
3. The method of claim 2, further comprising, after selecting the first server from the first set of servers:
and counting the total response message waiting amount, the response message waiting amount of the first server and the request sending amount of the first server.
4. The method of claim 3, wherein the selecting the first server from the first set of servers further comprises:
judging whether the request response detection duration of the first server to the request message needs to be counted or not;
and recording the sending time of the request message when the request response detection time is judged to need to be counted.
5. The method of claim 3, further comprising:
and when a response message to the request message returned from the first server is received, reducing the total response message waiting amount and the response message waiting amount of the first server.
6. The method of claim 5, wherein the method further comprises:
and when the response message is a successful response message, increasing the successful response quantity of the first server.
7. The method of claim 4, wherein the method further comprises:
when a response message to the request message returned from the first server is received, judging whether the request response detection duration of the first server to the request message needs to be counted or not;
and when the request response detection duration needs to be counted, calculating the request response detection duration of the first server according to the sending time of the request message and the time of receiving the response message.
8. The method according to any one of claims 1 to 7, wherein the method further comprises:
calculating the request sending quantity and the successful response quantity of each server in a preset time;
obtaining the ratio of the request sending quantity and the successful response quantity of each server in the preset time;
adding the server with the ratio larger than a preset threshold value into the shielding server set.
9. The method of claim 8, further comprising:
detecting each server in the shielding server set at regular time;
and when a server which is successful in establishing the communication link is detected, or a server which requests that the mean value of the response detection time length accords with the predetermined range of the normal distribution, or a server of which the ratio of the request sending quantity to the successful response quantity is smaller than the predetermined threshold value is detected, the shielding state of the server is released.
10. The method of claim 9, wherein the periodically detecting each server in the set of shadow servers comprises:
and sending request messages to all the servers in the shielding server set at regular time according to a preset frequency, and updating the value of the abnormal detection quantity of the servers.
11. A data reporting link monitoring device is characterized in that the device is applied to a production end which needs to report data to a server, and comprises:
the device comprises a selecting unit, a judging unit and a sending unit, wherein the selecting unit is used for selecting a first server from a first server set according to a preset selection rule when a request message is sent, and the first server set is a complement of a shielding server set in a total server set;
a statistics unit to:
requesting establishment of a communication link with the first server;
when a communication link is established with a first server, if the establishment of the communication link fails, increasing the request failure amount, and requesting the establishment of the communication link with the first server again, wherein the request failure amount is the number of times of the failure of the communication link establishment request from the first server by a production end within a preset time;
the link failure processing unit is used for adding the first server into the shielding server set when the request failure amount exceeds a preset value;
the link success processing unit is used for clearing the request failure amount of the first server when the request for establishing the communication link is successful;
the time length obtaining unit is used for obtaining the request response detection time length of each server in preset time; and
the second adding unit is used for adding the server with the request response detection duration being out of the preset range and the average value of the request response detection duration being maximum into the shielding server set according to normal distribution;
the apparatus is further configured to:
after the request failure amount of the first server is cleared, taking the first server out of the shielding server set;
and detecting each server in the shielding server set at regular time, and taking the first server out of the shielding server set when the mean value of the request response detection time length of the first server accords with the preset range selected according to the normal distribution.
12. An electronic device, comprising:
a processor; and
a memory having computer readable instructions stored thereon which, when executed by the processor, implement the method of data reporting link monitoring according to any one of claims 1 to 10.
13. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, implements a method for data reporting link monitoring according to any one of claims 1 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710329500.3A CN107204875B (en) | 2017-05-11 | 2017-05-11 | Data reporting link monitoring method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710329500.3A CN107204875B (en) | 2017-05-11 | 2017-05-11 | Data reporting link monitoring method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107204875A CN107204875A (en) | 2017-09-26 |
| CN107204875B true CN107204875B (en) | 2022-08-23 |
Family
ID=59905806
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710329500.3A Active CN107204875B (en) | 2017-05-11 | 2017-05-11 | Data reporting link monitoring method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107204875B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108122129B (en) * | 2017-12-01 | 2022-02-22 | 上海子午线新荣科技有限公司 | Data processing method and device and electronic equipment |
| CN109936613B (en) * | 2017-12-19 | 2021-11-05 | 北京京东尚科信息技术有限公司 | Disaster recovery method and device applied to server |
| CN108347350B (en) * | 2018-01-25 | 2022-04-15 | 中国银联股份有限公司 | Communication method and device |
| CN108551402B (en) * | 2018-04-09 | 2021-06-18 | 网易宝有限公司 | Information sending method and system, medium and computing equipment thereof |
| CN108712489A (en) * | 2018-05-11 | 2018-10-26 | 中国平安人寿保险股份有限公司 | A kind of breaking control method, device and its readable storage medium storing program for executing of network linking |
| CN109474489B (en) * | 2018-11-02 | 2021-04-20 | 新华三信息安全技术有限公司 | Link detection method, device and network equipment |
| CN113676365B (en) * | 2020-05-13 | 2022-10-11 | 北京达佳互联信息技术有限公司 | Access request processing method and device and electronic equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102075380A (en) * | 2010-12-16 | 2011-05-25 | 中兴通讯股份有限公司 | Method and device for detecting server state |
| CN103023938A (en) * | 2011-09-26 | 2013-04-03 | 阿里巴巴集团控股有限公司 | Service capability control method and system of server cluster |
| CN104065508A (en) * | 2014-06-23 | 2014-09-24 | 浪潮(北京)电子信息产业有限公司 | Application service health examination method, device and system |
| CN104618493A (en) * | 2015-02-12 | 2015-05-13 | 小米科技有限责任公司 | Data request processing method and device |
| CN106412091A (en) * | 2016-10-25 | 2017-02-15 | 广东欧珀移动通信有限公司 | Method, device and system for controlling data transmission |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4720295B2 (en) * | 2005-06-02 | 2011-07-13 | 日本電気株式会社 | Abnormality detection system and maintenance system |
| CN102333007B (en) * | 2011-09-28 | 2013-09-18 | 重庆大学 | On-line Web service quality monitoring system and method |
| CN104915846A (en) * | 2015-06-18 | 2015-09-16 | 北京京东尚科信息技术有限公司 | Electronic commerce time sequence data anomaly detection method and system |
| CN106547852B (en) * | 2016-10-19 | 2021-03-12 | 腾讯科技（深圳）有限公司 | Abnormal data detection method and device, and data preprocessing method and system |
-
2017
- 2017-05-11 CN CN201710329500.3A patent/CN107204875B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102075380A (en) * | 2010-12-16 | 2011-05-25 | 中兴通讯股份有限公司 | Method and device for detecting server state |
| CN103023938A (en) * | 2011-09-26 | 2013-04-03 | 阿里巴巴集团控股有限公司 | Service capability control method and system of server cluster |
| CN104065508A (en) * | 2014-06-23 | 2014-09-24 | 浪潮(北京)电子信息产业有限公司 | Application service health examination method, device and system |
| CN104618493A (en) * | 2015-02-12 | 2015-05-13 | 小米科技有限责任公司 | Data request processing method and device |
| CN106412091A (en) * | 2016-10-25 | 2017-02-15 | 广东欧珀移动通信有限公司 | Method, device and system for controlling data transmission |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107204875A (en) | 2017-09-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107204875B (en) | Data reporting link monitoring method and device, electronic equipment and storage medium | |
| US10904112B2 (en) | Automatic capture of detailed analysis information based on remote server analysis | |
| US8805999B2 (en) | Administering event reporting rules in a distributed processing system | |
| CN107370806B (en) | HTTP status code monitoring method, device, storage medium and electronic equipment | |
| CN109936613B (en) | Disaster recovery method and device applied to server | |
| CN110740072B (en) | Fault detection method, device and related equipment | |
| CN113268399B (en) | Alarm processing method and device and electronic equipment | |
| CN109039817B (en) | Information processing method, device, equipment and medium for flow monitoring | |
| CN110231998B (en) | Detection method and device for distributed timing task and storage medium | |
| CN109769029B (en) | Communication connection method based on electricity consumption information acquisition system and terminal equipment | |
| CN110717132A (en) | Data collection method and pushing method for full-link monitoring system and related equipment | |
| CN114090366A (en) | Method, device and system for monitoring data | |
| CN110708234A (en) | Message transmission processing method, message transmission processing device and storage medium | |
| CN113206797A (en) | Flow control method and device, electronic equipment and storage medium | |
| CN110225113B (en) | Service monitoring method and system | |
| US10616081B2 (en) | Application aware cluster monitoring | |
| CN112817687A (en) | Data synchronization method and device | |
| CN112994934B (en) | Data interaction method, device and system | |
| CN116260747A (en) | Monitoring method and device of terminal test equipment and electronic equipment | |
| CN113961641A (en) | Database synchronization method, device, equipment and storage medium | |
| US7908546B2 (en) | Methods and apparatus for detection of performance conditions in processing system | |
| CN110852537A (en) | Service quality detection method and device | |
| CN114500255B (en) | Log data reporting method, device, equipment and storage medium | |
| CN114064362B (en) | Data recovery method, system and computer readable storage medium for distributed storage | |
| CN112769889B (en) | Service data pushing method and device, storage medium and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |