CN107124445B - data acquisition method and device - Google Patents
data acquisition method and device Download PDFInfo
- Publication number
- CN107124445B CN107124445B CN201710211511.1A CN201710211511A CN107124445B CN 107124445 B CN107124445 B CN 107124445B CN 201710211511 A CN201710211511 A CN 201710211511A CN 107124445 B CN107124445 B CN 107124445B
- Authority
- CN
- China
- Prior art keywords
- data acquisition
- server
- adjustment coefficient
- data
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/16—Threshold monitoring
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1029—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers using data related to the state of servers by a load balancer
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
Abstract
the embodiment of the invention provides a data acquisition method and a data acquisition device, wherein a load state value of a server is acquired according to a preset period; acquiring a data acquisition adjustment coefficient of the current period according to the load state value and the corresponding relation between a preset load state value and the acquisition adjustment coefficient; the data acquisition adjustment coefficient of the current period is sent to the client, so that the client adjusts the frequency of data acquisition or sending according to the data acquisition adjustment coefficient of the current period, the frequency of data acquisition or sending of the client can be dynamically adapted to the load state of the server, and the adjustment can be carried out along with the change of the load state of the server, so that the frequency of data acquisition or sending can be reduced when the load of the server is high, and the data delay or loss caused by overlarge pressure of the server is avoided.
Description
Technical Field
the invention relates to the technical field of internet, in particular to a data acquisition method and a data acquisition device.
Background
With the rapid development of the internet, many new applications are developed and online every day, and users can obtain various services, such as a video playing client, a mobile communication client, and the like, through clients of various applications. The use of these clients has become an integral part of modern life.
In order to better improve the user experience when the user uses various application clients, the development or operation company of the application needs to fully understand various situations when the user uses various application clients. Therefore, the client needs to count various operation data when the user uses the client and send the data to the server for data analysis, so that problems can be found timely, or the client is further improved.
in the prior art, no matter whether the service of a server is busy or not, a client acquires and sends data to the server according to the natural frequency, when the service volume of the server is large, the processing capacity of the server is insufficient, the client may still acquire and send data to the server at a high frequency, and large data pressure is caused to the server, so that the server cannot timely receive the data sent by the client, and even the data is lost under extreme conditions.
Disclosure of Invention
the embodiment of the invention aims to provide a data acquisition method and a data acquisition device, so that the frequency of acquiring or sending data by a client is adjusted according to the current server traffic, the high-frequency acquisition of the client and the sending of the data to a server are avoided when the server traffic is large, the server can receive the data in time, the loss of the data is avoided, and the pressure of the server is reduced. The specific technical scheme is as follows:
The embodiment of the invention discloses a data acquisition method, which is applied to a server and comprises the following steps:
acquiring a load state value of the server according to a preset period, wherein the server is used for receiving data collected and sent by a client;
Acquiring a data acquisition adjustment coefficient of the current period according to the load state value and the corresponding relation between a preset load state value and the acquisition adjustment coefficient;
and sending the data acquisition adjustment coefficient of the current period to the client so that the client adjusts the frequency of data acquisition or sending according to the data acquisition adjustment coefficient of the current period.
Optionally, the obtaining the load state value of the server according to the preset period includes:
the method comprises the steps of obtaining average load state values of a plurality of slave servers according to a preset period, wherein the slave servers are subordinate to the servers and used for receiving data collected and sent by a plurality of clients in a distributed mode, and the average load state value is the average value of the load state values of the slave servers.
optionally, before obtaining the current average load state values of the plurality of slave servers according to the preset period, the method further includes:
and distributing a data volume with a difference value smaller than a preset value to each slave server through a load balancing strategy, wherein the data volume is the data volume of the data collected and sent by the plurality of clients.
Optionally, the load status value of the server includes a query rate per second of the server.
Optionally, before the obtaining the load state value of the server according to the preset period, the method further includes:
configuring an initial data acquisition adjustment coefficient as an initial value;
Correspondingly, the obtaining of the data acquisition adjustment coefficient of the current period according to the load state value and the corresponding relationship between the preset load state value and the acquisition adjustment coefficient includes:
When the query rate per second of the server is smaller than or equal to a preset threshold value and the data acquisition adjustment coefficient of the previous period of the current period is larger than or equal to an initial value, configuring the data acquisition adjustment coefficient of the current period as the initial value;
When the query rate per second is less than or equal to a preset threshold and the data acquisition adjustment coefficient of the previous period of the current period is less than an initial value, obtaining the data acquisition adjustment coefficient of the current period through a first preset formula, where the first preset formula is:
Z＝Z′+(X-Y)/Y
wherein Z represents the data acquisition adjustment coefficient of the current cycle, Z' represents the data acquisition adjustment coefficient of the last cycle, X represents a preset threshold, and Y represents the query rate per second;
when the query rate per second is greater than a preset threshold, obtaining the data acquisition adjustment coefficient of the current period according to a second preset formula, where the second preset formula is:
Z＝Z′-(X-Y)/Y
Wherein Z represents the data acquisition adjustment coefficient of the current period, Z' represents the data acquisition adjustment coefficient of the last period, X represents a preset threshold, and Y represents the query rate per second.
Optionally, the sending the data acquisition adjustment coefficient of the current period to the client includes:
and after receiving the data sent by the client, sending the data acquisition adjustment coefficient of the current period to the client.
The embodiment of the invention discloses a data acquisition method, which is applied to a client and comprises the following steps:
Receiving a data acquisition adjustment coefficient sent by a server;
and when the client side carries out data acquisition or transmission, adjusting the frequency of data acquisition or transmission according to the data acquisition adjustment coefficient, wherein the data comprises the operation data of the client side and is used for analyzing and monitoring the operation condition of the client side.
Optionally, the adjusting the frequency of data acquisition or transmission according to the data acquisition adjustment coefficient includes:
And merging or sampling the data acquired by the client through the data acquisition adjustment coefficient.
The embodiment of the invention discloses a data acquisition device, which is applied to a server and comprises:
the acquisition module is used for acquiring the load state value of the server according to a preset period, and the server is used for receiving data acquired and sent by the client;
The calculation module is used for acquiring a data acquisition adjustment coefficient of the current period according to the load state value and through the corresponding relation between a preset load state value and the acquisition adjustment coefficient;
And the sending module is used for sending the data acquisition adjustment coefficient of the current period to the client so as to enable the client to adjust the frequency of data acquisition or sending according to the data acquisition adjustment coefficient of the current period.
Optionally, the obtaining module is specifically configured to:
The method comprises the steps of obtaining average load state values of a plurality of slave servers according to a preset period, wherein the slave servers are subordinate to the servers and used for receiving data collected and sent by a plurality of clients in a distributed mode, and the average load state value is the average value of the load state values of the slave servers.
Optionally, the apparatus further comprises:
And the balancing module is used for distributing data volume with the difference value smaller than a preset value to each slave server through a load balancing strategy, wherein the data volume is the data volume of the data collected and sent by the plurality of clients.
Optionally, the load status value of the server in the apparatus comprises a query rate per second of the server.
optionally, the apparatus further comprises:
the initial module is used for configuring an initial data acquisition adjustment coefficient into an initial value;
Correspondingly, the computing module comprises:
the first sub-module is used for configuring the data acquisition adjustment coefficient of the current period as an initial value when the query rate per second of the server is less than or equal to a preset threshold value and the data acquisition adjustment coefficient of the previous period of the current period is greater than or equal to the initial value;
The second sub-module is configured to, when the query rate per second is less than or equal to a preset threshold and the data acquisition adjustment coefficient of the previous cycle of the current cycle is less than an initial value, obtain the data acquisition adjustment coefficient of the current cycle through a first preset formula, where the first preset formula is:
Z＝Z′+(X-Y)/Y
Wherein Z represents the data acquisition adjustment coefficient of the current cycle, Z' represents the data acquisition adjustment coefficient of the last cycle, X represents a preset threshold, and Y represents the query rate per second;
A third sub-module, configured to, when the query rate per second is greater than a preset threshold, obtain a data acquisition adjustment coefficient of the current period according to a second preset formula, where the second preset formula is:
Z＝Z′-(X-Y)/Y
wherein Z represents the data acquisition adjustment coefficient of the current period, Z' represents the data acquisition adjustment coefficient of the last period, X represents a preset threshold, and Y represents the query rate per second.
Optionally, the sending module is specifically configured to:
and after receiving the data sent by the client, sending the data acquisition adjustment coefficient of the current period to the client.
The embodiment of the invention discloses a data acquisition device, which is applied to a client and comprises:
The receiving module is used for receiving the data acquisition adjustment coefficient sent by the server;
And the adjusting module is used for adjusting the frequency of data acquisition or transmission according to the data acquisition adjusting coefficient when the client acquires or transmits data, wherein the data comprises the operation data of the client and is used for analyzing and monitoring the operation condition of the client.
optionally, the adjusting module is specifically configured to:
and merging or sampling the data acquired by the client through the data acquisition adjustment coefficient.
the data acquisition method and the data acquisition device provided by the embodiment of the invention can adjust the frequency of data acquisition or data transmission of the client according to the current traffic of the server. Therefore, when the server has larger service volume, the frequency of acquiring or sending data is reduced, the server can receive the data without time delay in time, the data loss is avoided, the frequency of acquiring or sending data is reduced when the server has larger pressure, and the pressure of the server can be reduced to a certain extent. Of course, it is not necessary for any product or method of practicing the invention to achieve all of the above-described advantages at the same time.
drawings
in order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic diagram of a data acquisition method applied to a server according to an embodiment of the present invention;
FIG. 2 is a diagram illustrating a server architecture according to an embodiment of the present invention;
FIG. 3 is a flow chart of calculating a data acquisition adjustment factor for a current cycle;
Fig. 4 is a flowchart of a data acquisition method applied to a client according to an embodiment of the present invention;
Fig. 5 is a structural diagram of a data acquisition device applied to a server according to an embodiment of the present invention;
fig. 6 is a structural diagram of a data acquisition device applied to a client according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
referring to fig. 1, fig. 1 is a schematic diagram of a data acquisition method applied to a server according to an embodiment of the present invention, including:
step 101, acquiring a load state value of a server according to a preset period, wherein the server is used for receiving data collected and sent by a client.
the server may obtain the load state value of the server according to a preset period, where the preset period may be flexibly set according to needs, for example, the preset period may be 30 seconds, 1 minute, or 5 minutes, and the server may obtain the load state value of the server every 30 seconds, 1 minute, or 5 minutes.
The load status value of the server is used to reflect the status of the operation load of the server, and can be represented by many existing parameters, such as query rate per second, CPU usage rate, memory occupancy rate, etc. of the server.
Specifically, the query rate per second of the server, the CPU utilization rate, the memory occupancy rate, and the like may be obtained directly from the server by using tools such as server pressure test software, and any manner in which the load state value may be obtained belongs to the protection scope of the embodiment of the present invention, and details are not described herein again.
And 102, acquiring a data acquisition adjustment coefficient of the current period according to the load state value and the corresponding relation between the preset load state value and the acquisition adjustment coefficient.
after the load state value is obtained, the data acquisition coefficient of the current period can be determined according to the load state value.
The data acquisition coefficient can be a unitless parameter and is used for enabling the client to adjust the frequency of acquiring or sending data according to the size of the parameter, so that the frequency of sending data to the server by the client is adaptive to the load state of the server, and the problems that the server is overloaded, the data is lost and the like due to the fact that the client still sends data to the server at a higher frequency under the conditions that the server load is very high and the traffic is very large are avoided.
in practical application, the value range of the data acquisition coefficient can be set to be greater than 0 and less than or equal to 1, and when the value is 1, the client does not need to adjust the frequency of acquiring or collecting data. When the data collection coefficient takes on other values, the client may reduce the frequency of collecting or collecting data according to the data collection coefficient, for example, when the data collection coefficient is 0.5, the frequency of collecting or collecting data may be reduced by 50%.
inside the server, a corresponding relationship between the load state value and the acquisition adjustment coefficient may be preset in advance, and the corresponding relationship may be in the form of a formula, a comparative table, or the like. For example, a comparison table of CPU occupancy rates and acquisition adjustment coefficients in the form of table 1 may be preset in the server, and the corresponding acquisition adjustment coefficients are directly obtained from table 1 through different CPU occupancy rates.
TABLE 1
| CPU occupancy rate | 90％～100％ | 70％～90％ | 50％～70％ | ≤50％ |
| collecting adjustment coefficients | 0.5 | 0.7 | 0.8 | 1 |
and 103, sending the data acquisition adjustment coefficient of the current period to the client, so that the client adjusts the frequency of data acquisition or sending according to the data acquisition adjustment coefficient of the current period.
The state of the operation load of the server changes with time, and it is impossible to operate in a high load or low load state all the time. Therefore, the server needs to update the data acquisition adjustment coefficient according to a preset period.
when a preset period starts, the server obtains a load state value, and obtains a data acquisition coefficient of the current period through the load state value. And sending the data acquisition coefficient of the current period to the client.
And after the client receives the data acquisition coefficient of the current period sent by the server, adjusting the data acquisition or sending frequency according to the data acquisition coefficient. And when the client receives the data acquisition coefficient of the next period again in the next period, adjusting the frequency of data acquisition or transmission again according to the newly received data acquisition coefficient.
in the embodiment of the invention, the server obtains the load state value according to the preset period, obtains the data acquisition coefficient of the current period through the load state value, and sends the data acquisition coefficient of the current period to the client, so that the client adjusts the data acquisition or transmission frequency according to the data acquisition coefficient of the current period. The frequency of data acquisition or transmission of the client can be dynamically adapted to the load state of the server, and can be adjusted along with the change of the load state of the server, so that the frequency of data acquisition or transmission can be reduced when the server has large service volume and high load, the server can timely receive data without delay, the data loss is avoided, the stability of data transmission is ensured, the frequency of data acquisition or transmission is reduced when the pressure of the server is high, the pressure of the server is further reduced, and the normal operation of the server is ensured.
optionally, in the data acquisition method provided in the embodiment of the present invention, acquiring the load state value of the server according to a preset period includes:
the method comprises the steps of obtaining average load state values of a plurality of slave servers according to a preset period, wherein the slave servers are subordinate to the servers and used for receiving data collected and sent by a plurality of clients in a distributed mode, and the average load state value is the average value of the load state values of the plurality of slave servers. The server acts as a master for the slave servers.
in practical applications, it is difficult to use one server to receive data sent by all clients because of the huge number of clients. To cope with the huge amount of data, a distributed server system may be employed to receive data sent by a large number of clients.
The distributed server generally comprises a Master server and a plurality of slave servers, wherein the Master server and the slave servers are slave servers, the slave servers are subordinate to the Master server, and the Master server can uniformly manage and schedule the slave servers.
Each slave server can receive data sent by the client, the running load state of each slave server can also change along with time, and the master server can regularly acquire the running state values of each slave server according to a preset period and calculate the average running state value of all the slave servers. And carrying out subsequent steps according to the average running state value.
In the implementation of the invention, the framework of the master server and the slave server is used to ensure that a large amount of data sent by the client can be received, the server is prevented from being crushed due to excessive data, and the service quality and the stability are improved.
Optionally, in the data acquisition method provided in the embodiment of the present invention, before obtaining the current average load state values of the plurality of slave servers according to the preset period, the method further includes:
and distributing the data volume of which the difference value is smaller than the preset value to each slave server through a load balancing strategy, wherein the data volume is the data volume of the data collected and sent by the plurality of clients.
Referring to fig. 2, fig. 2 is a schematic diagram of a server architecture according to an embodiment of the present invention.
fig. 2 shows that a plurality of clients include: client a201, client B202, client C203. The clients respectively collect or send data, and when each client sends data, the clients can uniformly distribute the data sent by the client a201, the client B202 and the client C203 to the slave server a205, the slave server B206 and the slave server C207 by applying a corresponding load balancing strategy through the load balancing device 204. The master server 208 may obtain the load status value of each slave server at a preset period.
Specifically, the load balancing device may be integrated in a router, or may be formed by a dedicated server, and in the prior art, many methods and applications exist for the load balancing device and a corresponding load balancing policy, which may be selected as needed, and are not described herein again.
Optionally, in the data acquisition method provided in the embodiment of the present invention, the load state value of the server includes a query rate per second of the server.
among the various parameters that may be used to represent the load status value, the server's query rate per second may be used to represent the load status value of the server.
QPS (Query Per Second) is a measure of how much traffic a particular Query server processes within a specified time, and on the internet, the performance of a machine as a server is often measured by the Query Per Second, i.e., the number of requests Per Second.
The load state value of the server is represented by the query rate per second, and the query rate per second is easier to quantify, so that the operation load of the server can be more accurately represented, and the query rate per second can be a very specific numerical value, so that the data acquisition coefficient can be more accurately determined by the query rate per second through a formula or a function.
optionally, in the data acquisition method provided in the embodiment of the present invention, before the load state value of the server is obtained according to the preset period, the method further includes:
and configuring the initial data acquisition adjustment coefficient as an initial value.
Before the server starts to acquire the load state value of the server, an initial data acquisition adjustment coefficient may be initialized and configured as an initial value, where the initial value may indicate that the client does not need to adjust the frequency of data acquisition or transmission, that is, the client performs data acquisition or transmission at a normal frequency. The initial value may be generally expressed by an integer, and it is preferable in the embodiment of the present invention to configure the initial value to 1.
Correspondingly, when the query rate per second is used to represent the load state value of the server, the obtaining of the data acquisition adjustment coefficient in the current period according to the load state value and the corresponding relationship between the preset load state value and the acquisition adjustment coefficient includes:
In the first case, when the query rate per second of the server is less than or equal to the preset threshold and the data acquisition adjustment coefficient of the previous cycle of the current cycle is greater than or equal to the initial value, the data acquisition adjustment coefficient of the current cycle is configured as the initial value.
the preset threshold is a threshold that can be set according to needs, when the query rate per second is greater than the preset threshold, the operation load of the server is considered to be high, and the traffic is large, and when the query rate per second is less than or equal to the preset threshold, the operation load of the server is considered to be normal, and the traffic is not large.
The server acquires the query rate per second according to a preset period, timely adjusts the data acquisition coefficient of the current period according to the query rate per second, when the query rate per second of the current period is smaller than or equal to a preset threshold value, the operation load of the server of the current period can be considered to be in a normal level, and if the data acquisition adjustment coefficient of the previous period of the current period is larger than or equal to an initial value, the data acquisition adjustment coefficient of the current period needs to be readjusted to the initial value, which means that the client can acquire or send data according to a normal frequency.
under the second condition, when the query rate per second is less than or equal to the preset threshold and the data acquisition adjustment coefficient of the previous period of the current period is less than the initial value, the data acquisition adjustment coefficient of the current period is obtained through a first preset formula, wherein the first preset formula is as follows:
Z＝Z′+(X-Y)/Y
Wherein Z represents the data acquisition adjustment coefficient of the current period, Z' represents the data acquisition adjustment coefficient of the last period, X represents a preset threshold value, and Y represents the query rate per second.
The query rate per second of the current period is less than or equal to the preset threshold, and when the data acquisition adjustment coefficient of the previous period of the current period is less than the initial value, the server operation load in the current period can be considered to be in a normal state, but because the data acquisition adjustment coefficient of the previous period is less than the initial value, the server operation load in the previous period is very high, and if the data acquisition adjustment coefficient is immediately restored to the initial value, the server operation load may be increased sharply, so that the data acquisition adjustment coefficient of the current period can be increased on the basis of the data acquisition adjustment coefficient of the previous period through the first preset formula, and is closer to the initial value. Of course, there may be a case where the adjusted data collection adjustment coefficient exceeds the initial value, and in this case, the server may restore the data collection adjustment coefficient to the initial value in the next cycle according to the first case described above.
Under the third condition, when the query rate per second is greater than the preset threshold, obtaining the data acquisition adjustment coefficient of the current period according to a second preset formula, wherein the second preset formula is as follows:
Z＝Z′-(X-Y)/Y
wherein Z represents the data acquisition adjustment coefficient of the current period, Z' represents the data acquisition adjustment coefficient of the last period, X represents a preset threshold value, and Y represents the query rate per second.
When the query rate per second is greater than the preset threshold, it indicates that the operation load of the server in the current period is in an excessively high state, and the data acquisition adjustment coefficient in the current period can be reduced on the basis of the data acquisition adjustment coefficient in the previous period according to a second preset formula, so that the frequency of data acquisition or transmission is further reduced by the client.
Referring to fig. 3, fig. 3 is a flow chart of calculating a data acquisition adjustment factor for a current cycle, including:
step 301, initializing data acquisition adjustment coefficient, and adjusting data acquisition adjustment coefficient Z0The configuration is 1.
step 302, obtaining the query rate per second of each slave server according to a preset period, and calculating an average query rate per second Y.
Step 303, determining whether the average query rate per second is less than or equal to a preset threshold value X, i.e., determining whether Y is less than or equal to X.
and 304, when Y is less than or equal to X, judging whether the data acquisition adjustment coefficient Z' in the previous period is greater than or equal to the initial value 1.
In step 305, when the data acquisition coefficient Z' in the previous period is greater than the initial value 1, the data acquisition adjustment coefficient in the current period is configured to be 1, that is, Z is 1.
And step 306, when Y is less than or equal to X and the data acquisition adjusting coefficient Z 'in the previous period is less than 1, the data acquisition adjusting coefficient Z in the current period is Z' + (X-Y)/Y, and when the calculated Z is less than 0.1, Z is configured to be 0.1.
And 307, when the average query rate per second is greater than a preset threshold value, namely Y > X, setting the data acquisition adjustment coefficient Z of the current period to be Z' - (X-Y)/Y, and when the calculated Z is less than 0.1, configuring Z to be 0.1.
And 308, sending the data acquisition adjustment coefficient of the current period to the client. And after the data acquisition adjustment coefficient of the current period is obtained, the data acquisition adjustment coefficient can be sent to each client, so that each client can adjust the frequency of data acquisition or sending according to the data acquisition adjustment coefficient.
and after the transmission is finished, when the next preset period is reached, the server acquires the query rate per second of each slave server again, and calculates the data acquisition adjustment coefficient again.
Optionally, in the data acquisition method provided in the embodiment of the present invention, sending the data acquisition adjustment coefficient of the current period to the client includes:
And after receiving the data sent by the client, sending the data acquisition adjustment coefficient of the current period to the client.
In practical application, the server does not necessarily need to send the data acquisition adjustment coefficient to the client at each preset period, and the data acquisition adjustment coefficient can be sent to the client after the client sends data to the server.
The client sends data to the server, which indicates that the client is in a working state and is collecting and sending the data. When the server receives the data sent by the client, the data acquisition adjustment coefficient is sent to the client, so that the frequency of sending the data to the client can be reduced, and the workload of the server is reduced. And the transmission to the client which is not in the working state is avoided, and the resource waste is reduced.
Referring to fig. 4, fig. 4 is a flowchart of a data acquisition method applied to a client according to an embodiment of the present invention. The method comprises the following steps:
Step 401, receiving a data acquisition adjustment coefficient sent by a server.
The client can receive the data acquisition adjustment coefficient sent by the server according to a certain period, and the frequency of data acquisition or sending is adjusted according to the latest received data acquisition adjustment coefficient when the client receives the data acquisition adjustment coefficient once.
Step 402, when the client performs data acquisition or transmission, adjusting the frequency of data acquisition or transmission according to the data acquisition adjustment coefficient, wherein the data comprises the operation data of the client and is used for analyzing and monitoring the operation condition of the client.
After receiving the data acquisition adjustment coefficient, the client can adjust the frequency of data acquisition or transmission according to the data acquisition adjustment coefficient. The frequency of data acquisition can be reduced according to the data acquisition adjustment coefficient, or the frequency of data transmission to the server can be reduced according to the data acquisition adjustment coefficient.
For example, the default data collection frequency of the client is to collect data 10 times every 1 minute, and when the received data collection adjustment coefficient is 0.5, the frequency of data collection may be reduced by 0.5 times, and data collection may be performed 5 times for 1 minute. Similarly, the transmission frequency of the data can be reduced according to the corresponding method.
In the embodiment of the invention, when the client side collects or sends the data, the frequency of the collected or sent data is adjusted through the data collection adjustment coefficient sent by the server, so that the frequency of the data collection or sending is reduced, the quantity and the sending times of the sent data are reduced, the pressure of the server is reduced when the operation load of the server is higher, and the data loss is avoided.
Optionally, in the data acquisition method provided in the embodiment of the present invention, adjusting the frequency of data acquisition or transmission according to the data acquisition adjustment coefficient includes:
And merging or sampling the data acquired by the client through the data acquisition adjustment coefficient.
In practical application, data merging or data sampling can be performed on data which can be acquired by a client according to the data acquisition adjustment coefficient, so that the data volume of the data is reduced. Data merging is to merge a plurality of pieces of data into one piece of data, and the data is easier to compress after being merged, so that the data volume is reduced. The data sampling is to sample the original data in a certain proportion and discard a part of the data, thereby directly reducing the data volume.
for example, when the data collection adjustment coefficient is 0.9, the raw data may be sampled at a rate of 90%, so as to directly reduce the data amount by 10%, and the data may be merged at a rate of 90%, of course, a plurality of rules using the data collection adjustment coefficient may be set on the client, and in the above example, when the data collection adjustment coefficient is 0.9, the data may be merged at a rate of 1-0.9 — 0.1, that is, 10% of the collected raw data may be merged. Specifically, merging or sampling data belongs to the prior art in the field of data processing, and is not described herein again.
By merging and sampling data, the amount of data can be further reduced, and the pressure on the server can be further reduced when data is transmitted to the server.
Referring to fig. 5, fig. 5 is a structural diagram of a data acquisition device applied to a server according to an embodiment of the present invention, including:
the obtaining module 501 is configured to obtain a load state value of a server according to a preset period, where the server is configured to receive data collected and sent by a client.
The calculating module 502 is configured to obtain a data acquisition adjustment coefficient of the current period according to the load state value and through a corresponding relationship between a preset load state value and the acquisition adjustment coefficient.
The sending module 503 is configured to send the data acquisition adjustment coefficient of the current period to the client, so that the client adjusts the frequency of data acquisition or sending according to the data acquisition adjustment coefficient of the current period.
in the embodiment of the invention, the server obtains the load state value according to the preset period, obtains the data acquisition coefficient of the current period through the load state value, and sends the data acquisition coefficient of the current period to the client, so that the client adjusts the data acquisition or transmission frequency according to the data acquisition coefficient of the current period. The frequency of data acquisition or transmission of the client can be dynamically adapted to the load state of the server, and can be adjusted along with the change of the load state of the server, so that the frequency of data acquisition or transmission can be reduced when the server has large service volume and high load, the server can timely receive data without delay, the data loss is avoided, the stability of data transmission is ensured, the frequency of data acquisition or transmission is reduced when the pressure of the server is high, the pressure of the server is further reduced, and the normal operation of the server is ensured.
The device of the embodiment of the invention is a device applying the data acquisition method, so that all the embodiments of the data acquisition method are suitable for the device and can achieve the same or similar beneficial effects.
Optionally, the obtaining module 501 is specifically configured to:
the method comprises the steps of obtaining average load state values of a plurality of slave servers according to a preset period, wherein the slave servers are subordinate to the servers and used for receiving data collected and sent by a plurality of clients in a distributed mode, and the average load state value is the average value of the load state values of the plurality of slave servers.
Optionally, the apparatus further comprises:
and the balancing module is used for distributing the data volume of which the difference value is smaller than the preset value to each slave server through a load balancing strategy, wherein the data volume is the data volume of the data collected and sent by the plurality of clients.
optionally, the load status value of the server in the apparatus comprises a query rate per second of the server.
Optionally, the apparatus further comprises:
And the initial module is used for configuring the initial data acquisition adjustment coefficient into an initial value.
Accordingly, the calculation module 502 includes:
The first submodule is used for configuring the data acquisition adjustment coefficient of the current period as an initial value when the query rate per second of the server is smaller than or equal to a preset threshold value and the data acquisition adjustment coefficient of the previous period of the current period is larger than or equal to the initial value.
the second sub-module is used for obtaining the data acquisition adjustment coefficient of the current period through a first preset formula when the query rate per second is less than or equal to a preset threshold and the data acquisition adjustment coefficient of the previous period of the current period is less than an initial value, and the first preset formula is as follows:
Z＝Z′+(X-Y)/Y
Wherein Z represents the data acquisition adjustment coefficient of the current period, Z' represents the data acquisition adjustment coefficient of the last period, X represents a preset threshold value, and Y represents the query rate per second.
the third sub-module is used for obtaining the data acquisition adjustment coefficient of the current period according to a second preset formula when the query rate per second is greater than a preset threshold, wherein the second preset formula is as follows:
Z＝Z′-(X-Y)/Y
wherein Z represents the data acquisition adjustment coefficient of the current period, Z' represents the data acquisition adjustment coefficient of the last period, X represents a preset threshold value, and Y represents the query rate per second.
optionally, the sending module 503 is specifically configured to:
And after receiving the data sent by the client, sending the data acquisition adjustment coefficient of the current period to the client.
Referring to fig. 6, fig. 6 is a structural diagram of a data acquisition device applied to a client according to an embodiment of the present invention, including:
The receiving module 601 is configured to receive a data acquisition adjustment coefficient sent by a server.
the adjusting module 602 is configured to adjust a frequency of data acquisition or data transmission according to a data acquisition adjustment coefficient when the client performs data acquisition or data transmission, where the data includes operation data of the client and is used to analyze and monitor an operation condition of the client.
In the embodiment of the invention, when the client side collects or sends the data, the frequency of the collected or sent data is adjusted through the data collection adjustment coefficient sent by the server, so that the frequency of the data collection or sending is reduced, the quantity and the sending times of the sent data are reduced, the pressure of the server is reduced when the operation load of the server is higher, and the data loss is avoided.
Optionally, the adjusting module 602 is specifically configured to:
And merging or sampling the data acquired by the client through the data acquisition adjustment coefficient.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
the above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (8)
1. A data acquisition method is applied to a server and comprises the following steps:
Configuring an initial data acquisition adjustment coefficient as an initial value;
acquiring a load state value of the server according to a preset period, wherein the server is used for receiving data acquired and sent by a client, and the load state value of the server comprises the query rate of the server per second;
when the query rate per second of the server is smaller than or equal to a preset threshold value and the data acquisition adjustment coefficient of the previous period of the current period is larger than or equal to the initial value, configuring the data acquisition adjustment coefficient of the current period as the initial value;
when the query rate per second is less than or equal to the preset threshold and the data acquisition adjustment coefficient of the previous period of the current period is less than the initial value, obtaining the data acquisition adjustment coefficient of the current period through a first preset formula, where the first preset formula is:
Z＝Z′+(X-Y)/Y
Wherein Z represents the data acquisition adjustment coefficient of the current period, Z' represents the data acquisition adjustment coefficient of the previous period, X represents a preset threshold, and Y represents the query rate per second;
when the query rate per second is greater than the preset threshold, obtaining the data acquisition adjustment coefficient of the current period according to a second preset formula, where the second preset formula is:
Z＝Z′-(X-Y)/Y；
and sending the data acquisition adjustment coefficient of the current period to the client so that the client adjusts the frequency of data acquisition or sending according to the data acquisition adjustment coefficient of the current period.
2. the method according to claim 1, wherein the obtaining the load status value of the server according to the preset period comprises:
The method comprises the steps of obtaining average load state values of a plurality of slave servers according to a preset period, wherein the slave servers are subordinate to the servers and used for receiving data collected and sent by a plurality of clients in a distributed mode, and the average load state value is the average value of the load state values of the slave servers.
3. The method according to claim 2, wherein before the obtaining the current average load status values of the plurality of slave servers according to the preset period, the method further comprises:
And distributing a data volume with a difference value smaller than a preset value to each slave server through a load balancing strategy, wherein the data volume is the data volume of the data collected and sent by the plurality of clients.
4. The method of claim 1, wherein sending the current period of data acquisition adjustment factor to the client comprises:
And after receiving the data sent by the client, sending the data acquisition adjustment coefficient of the current period to the client.
5. A data acquisition device, applied to a server, comprising:
the initial module is used for configuring an initial data acquisition adjustment coefficient into an initial value;
The server is used for receiving data collected and sent by a client, and the load state value of the server comprises the query rate of the server per second;
A computing module, comprising: a first sub-module, a second sub-module and a third sub-module,
the first sub-module is configured to configure the data acquisition adjustment coefficient of the current period as the initial value when the query rate per second of the server is less than or equal to a preset threshold and the data acquisition adjustment coefficient of the previous period of the current period is greater than or equal to the initial value;
The second sub-module is configured to, when the query rate per second is less than or equal to the preset threshold and the data acquisition adjustment coefficient of the previous cycle of the current cycle is less than the initial value, obtain the data acquisition adjustment coefficient of the current cycle through a first preset formula, where the first preset formula is:
Z＝Z′+(X-Y)/Y
Wherein Z represents the data acquisition adjustment coefficient of the current period, Z' represents the data acquisition adjustment coefficient of the previous period, X represents a preset threshold, and Y represents the query rate per second;
The third sub-module is configured to, when the query rate per second is greater than the preset threshold, obtain a data acquisition adjustment coefficient of the current period according to a second preset formula, where the second preset formula is:
Z＝Z′-(X-Y)/Y；
and the sending module is used for sending the data acquisition adjustment coefficient of the current period to the client so as to enable the client to adjust the frequency of data acquisition or sending according to the data acquisition adjustment coefficient of the current period.
6. The apparatus of claim 5, wherein the obtaining module is specifically configured to:
The method comprises the steps of obtaining average load state values of a plurality of slave servers according to a preset period, wherein the slave servers are subordinate to the servers and used for receiving data collected and sent by a plurality of clients in a distributed mode, and the average load state value is the average value of the load state values of the slave servers.
7. The apparatus of claim 6, further comprising:
And the balancing module is used for distributing data volume with the difference value smaller than a preset value to each slave server through a load balancing strategy, wherein the data volume is the data volume of the data collected and sent by the plurality of clients.
8. the apparatus of claim 5, wherein the sending module is specifically configured to:
and after receiving the data sent by the client, sending the data acquisition adjustment coefficient of the current period to the client.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710211511.1A CN107124445B (en) | 2017-03-31 | 2017-03-31 | data acquisition method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710211511.1A CN107124445B (en) | 2017-03-31 | 2017-03-31 | data acquisition method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107124445A CN107124445A (en) | 2017-09-01 |
| CN107124445B true CN107124445B (en) | 2019-12-13 |
Family
ID=59725660
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710211511.1A Active CN107124445B (en) | 2017-03-31 | 2017-03-31 | data acquisition method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107124445B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP4167094A4 (en) * | 2020-07-12 | 2023-11-22 | Huawei Technologies Co., Ltd. | Data acquisition method and device |
Families Citing this family (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107864071B (en) * | 2017-11-02 | 2021-06-22 | 江苏物联网研究发展中心 | Active safety-oriented dynamic data acquisition method, device and system |
| CN110635961B (en) * | 2018-06-22 | 2022-08-09 | 北京京东尚科信息技术有限公司 | Pressure measurement method, device and system of server |
| JP7010171B2 (en) * | 2018-08-10 | 2022-01-26 | 日本電信電話株式会社 | Maintenance management system and data processing method |
| CN110035055B (en) * | 2019-02-19 | 2022-02-01 | 中国铁建重工集团股份有限公司 | Method for transmitting remote data of industrial equipment |
| CN111291252B (en) * | 2020-02-21 | 2024-03-08 | 北京奇艺世纪科技有限公司 | Method and device for adjusting query rate per second, electronic equipment and storage medium |
| CN111813510A (en) * | 2020-06-09 | 2020-10-23 | 四川虹美智能科技有限公司 | Method, device and system for uploading data by intelligent equipment |
| CN111882382A (en) * | 2020-06-30 | 2020-11-03 | 上海万位科技有限公司 | Shared automobile data storage management method based on data and risk association model |
| CN111930079A (en) * | 2020-06-30 | 2020-11-13 | 西门子工厂自动化工程有限公司 | Data acquisition method, device and system, electronic equipment and storage medium |
| CN112491837B (en) * | 2020-11-17 | 2022-09-06 | 北京航空航天大学杭州创新研究院 | Safety control method and device based on industrial internet equipment state |
| CN112422349B (en) * | 2020-12-03 | 2023-04-07 | 广州瀚信通信科技股份有限公司 | Network management system, method, equipment and medium for NFV |
| CN112583669B (en) * | 2020-12-11 | 2022-06-24 | 北京字节跳动网络技术有限公司 | On-line link pressure testing method and device, electronic equipment and medium |
| CN112787850B (en) * | 2020-12-28 | 2023-05-09 | 北京金山云网络技术有限公司 | Method, system, device and storage medium for adjusting query rate per second |
| CN114844894A (en) * | 2022-04-29 | 2022-08-02 | 济南浪潮数据技术有限公司 | Data transmission method, device and medium for server node |
| CN117250928B (en) * | 2023-10-24 | 2024-04-12 | 河北燃气有限公司 | SCADA system data acquisition period control method and system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104168207A (en) * | 2014-08-21 | 2014-11-26 | 北京奇艺世纪科技有限公司 | Load regulating method and system |
| CN104423529A (en) * | 2013-08-30 | 2015-03-18 | 华为技术有限公司 | CPU (central processing unit) state adjustment method and device |
| CN105024933A (en) * | 2014-04-22 | 2015-11-04 | 腾讯科技（深圳）有限公司 | Request packet sending frequency control method and apparatus |
| CN105760467A (en) * | 2016-02-05 | 2016-07-13 | 浪潮(北京)电子信息产业有限公司 | Real-time data read-write method, device and system and NFS server |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP6092546B2 (en) * | 2012-08-22 | 2017-03-08 | キヤノン株式会社 | Data distribution system and control method thereof |
-
2017
- 2017-03-31 CN CN201710211511.1A patent/CN107124445B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104423529A (en) * | 2013-08-30 | 2015-03-18 | 华为技术有限公司 | CPU (central processing unit) state adjustment method and device |
| CN105024933A (en) * | 2014-04-22 | 2015-11-04 | 腾讯科技（深圳）有限公司 | Request packet sending frequency control method and apparatus |
| CN104168207A (en) * | 2014-08-21 | 2014-11-26 | 北京奇艺世纪科技有限公司 | Load regulating method and system |
| CN105760467A (en) * | 2016-02-05 | 2016-07-13 | 浪潮(北京)电子信息产业有限公司 | Real-time data read-write method, device and system and NFS server |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP4167094A4 (en) * | 2020-07-12 | 2023-11-22 | Huawei Technologies Co., Ltd. | Data acquisition method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107124445A (en) | 2017-09-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107124445B (en) | data acquisition method and device | |
| CN109194984B (en) | Video frame scheduling method based on edge calculation | |
| CN109962856B (en) | Resource allocation method, device and computer readable storage medium | |
| CN107171839B (en) | Bandwidth flow cost control method | |
| CN110740146B (en) | Method and device for scheduling cache nodes and computer network system | |
| CN110896357B (en) | Flow prediction method, device and computer readable storage medium | |
| CN107277013B (en) | Live broadcast system in peer-to-peer network and node management method | |
| CN110784929B (en) | Access resource allocation method, device, equipment and system | |
| CN107515571B (en) | Monitoring data acquisition method, device and system | |
| CN110716808A (en) | Service processing method, device, computer equipment and storage medium | |
| CN106464733A (en) | Method and device for adjusting virtual resources in cloud computing | |
| CN110012324B (en) | Code rate self-adaption method, WIFI camera, control device and code rate self-adaption system for embedded wireless video transmission | |
| US8599684B1 (en) | Congestion estimation | |
| CN109067607B (en) | Method and device for adjusting rated bandwidth | |
| WO2019029729A1 (en) | Method and device for reporting network performance parameters | |
| CN105791144A (en) | Method and apparatus for sharing link traffic | |
| Biernacki | Improving quality of adaptive video by traffic prediction with (F) ARIMA models | |
| CN114466194A (en) | Video coding adjusting method and device, storage medium and electronic equipment | |
| CN108664321B (en) | System resource allocation adjusting method and device | |
| CN111865817A (en) | Load balancing control method, device and equipment for remote measuring collector and storage medium | |
| CN109560978B (en) | Network flow detection method, device and system and computer readable storage medium | |
| JP6186303B2 (en) | Traffic amount upper limit prediction apparatus, method and program | |
| WO2015011220A1 (en) | Monitoring network use of resources | |
| CN106940673A (en) | One kind monitoring item interval adjustment method and system | |
| CN110543509A (en) | Monitoring system, method and device for user access data and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |