CN110838888A - Data processing method, device, system and computer readable storage medium - Google Patents
Data processing method, device, system and computer readable storage medium Download PDFInfo
- Publication number
- CN110838888A CN110838888A CN201810931533.XA CN201810931533A CN110838888A CN 110838888 A CN110838888 A CN 110838888A CN 201810931533 A CN201810931533 A CN 201810931533A CN 110838888 A CN110838888 A CN 110838888A
- Authority
- CN
- China
- Prior art keywords
- data processing
- data
- data receiving
- value
- receiving point
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 21
- 238000000034 method Methods 0.000 claims abstract description 28
- 230000008569 process Effects 0.000 claims abstract description 14
- 238000004590 computer program Methods 0.000 claims description 10
- 230000005540 biological transmission Effects 0.000 abstract description 4
- 238000010586 diagram Methods 0.000 description 18
- 238000012544 monitoring process Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 241000282414 Homo sapiens Species 0.000 description 1
- 238000009825 accumulation Methods 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000011664 signaling Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L1/00—Arrangements for detecting or preventing errors in the information received
- H04L1/0001—Systems modifying transmission characteristics according to link quality, e.g. power backoff
- H04L1/0036—Systems modifying transmission characteristics according to link quality, e.g. power backoff arrangements specific to the receiver
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L1/00—Arrangements for detecting or preventing errors in the information received
- H04L1/0001—Systems modifying transmission characteristics according to link quality, e.g. power backoff
- H04L1/0015—Systems modifying transmission characteristics according to link quality, e.g. power backoff characterised by the adaptation strategy
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0805—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability
- H04L43/0817—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability by checking functioning
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0876—Network utilisation, e.g. volume of load or congestion level
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0876—Network utilisation, e.g. volume of load or congestion level
- H04L43/0894—Packet rate
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/16—Threshold monitoring
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Environmental & Geological Engineering (AREA)
- Quality & Reliability (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a data processing method, a device, a system and a computer readable storage medium, and relates to the technical field of network transmission. The data processing method comprises the following steps: acquiring a data processing pressure value of a data receiving point; comparing the data processing pressure value with a preset threshold value; and adjusting the data receiving interval of the data receiving point according to the comparison result so that the data receiving point intercepts and processes the data in batches according to the adjusted data receiving interval. The embodiment of the invention can dynamically adjust the data receiving interval of the data receiving point, namely the time window value according to the data processing pressure, thereby enabling the set data receiving interval to be more matched with the actual requirement, and improving the stability of data processing and the resource utilization rate.
Description
Technical Field
The present invention relates to the field of network transmission technologies, and in particular, to a data processing method, apparatus, system, and computer-readable storage medium.
Background
In the current real-time data receiving, processing and dispatching process, a time window value is generally set at a data receiving point for intercepting data in a unit time batch for real-time processing. The time window value is adjusted by adopting a manual setting mode. However, the inaccuracy of the time window value set by human beings may reduce the stability and resource utilization rate during the data receiving and processing process.
Disclosure of Invention
The embodiment of the invention aims to solve the technical problem that: how to improve the stability and the resource utilization rate in the data receiving and processing process.
According to a first aspect of some embodiments of the present invention, there is provided a data processing method comprising: acquiring a data processing pressure value of a data receiving point; comparing the data processing pressure value with a preset threshold value; and adjusting the data receiving interval of the data receiving point according to the comparison result so that the data receiving point intercepts and processes the data in batches according to the adjusted data receiving interval.
In some embodiments, in the event that the data processing pressure value is greater than a preset high threshold, the data reception interval is narrowed; and under the condition that the data processing pressure value is smaller than the preset low threshold value, increasing the data receiving interval.
In some embodiments, in the case that the data processing pressure values are all greater than the preset high threshold value within the preset duration of time, the data reception interval is narrowed; and under the condition that the data processing pressure values are all smaller than a preset low threshold value within the continuous preset time, increasing the data receiving interval.
In some embodiments, further comprising: and acquiring a network pressure index value and a data receiving point pressure index value so as to determine a data processing pressure value according to the acquired network pressure index value and the acquired data receiving point pressure index value.
In some embodiments, the network pressure indicator includes at least one of flow and flow rate of the network, and the data sink pressure indicator includes at least one of CPU utilization, memory utilization, and data processing time.
In some embodiments, obtaining the data processing pressure values for the data receiving points comprises: and determining the data processing pressure value according to the weighted values of the various data processing pressure index values.
In some embodiments, the data received by the data receiving point is streaming data.
According to a second aspect of some embodiments of the present invention, there is provided a data processing apparatus comprising: the acquisition module is configured to acquire a data processing pressure value of the data receiving point; a comparison module configured to compare the data processing pressure value with a preset threshold; and the adjusting module is configured to adjust the data receiving interval of the data receiving point according to the comparison result, so that the data receiving point intercepts and processes the data in batches according to the adjusted data receiving interval.
In some embodiments, the adjustment module is further configured to narrow the data reception interval if the data processing pressure value is greater than a preset high threshold; and under the condition that the data processing pressure value is smaller than the preset low threshold value, increasing the data receiving interval.
In some embodiments, the adjustment module is further configured to narrow the data reception interval if the data processing pressure values are greater than the preset high threshold value for the preset duration of time; and under the condition that the data processing pressure values are all smaller than a preset low threshold value within the continuous preset time, increasing the data receiving interval.
In some embodiments, the data processing apparatus further comprises: an acquisition module configured to acquire a network pressure index value and a data receiving point pressure index value; the acquisition module is further configured to determine a data processing pressure value from the collected network pressure index value and the data receiving point pressure index value.
In some embodiments, the obtaining module is further configured to determine the data processing pressure value based on a weighted value of the plurality of data processing pressure indicator values.
According to a third aspect of some embodiments of the present invention there is provided a data processing system comprising: any one of the foregoing data processing apparatus; and a data receiving point configured to intercept and process the data in batches according to the data receiving interval adjusted by the data processing device.
According to a fourth aspect of some embodiments of the present invention, there is provided a data processing apparatus comprising: a memory; and a processor coupled to the memory, the processor configured to perform any of the foregoing data processing methods based on instructions stored in the memory.
According to a fifth aspect of some embodiments of the present invention, there is provided a computer readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements any one of the data processing methods described above.
Some embodiments of the above invention have the following advantages or benefits: the embodiment of the invention can dynamically adjust the data receiving interval of the data receiving point, namely the time window value according to the data processing pressure, thereby enabling the set data receiving interval to be more matched with the actual requirement, and improving the stability of data processing and the resource utilization rate.
Other features of the present invention and advantages thereof will become apparent from the following detailed description of exemplary embodiments thereof, which proceeds with reference to the accompanying drawings.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a flow diagram of a data processing method according to some embodiments of the invention.
Fig. 2 is a flow chart illustrating a method for adjusting a data reception interval according to some embodiments of the invention.
Fig. 3 is a flowchart illustrating a data reception interval adjustment method according to another embodiment of the present invention.
FIG. 4 is a block diagram of a data processing apparatus according to some embodiments of the invention.
FIG. 5 is a block diagram of a data processing system according to some embodiments of the present invention.
FIG. 6 is a block diagram of a data processing apparatus according to further embodiments of the present invention.
FIG. 7 is a block diagram of a data processing apparatus according to further embodiments of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The relative arrangement of the components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.
Meanwhile, it should be understood that the sizes of the respective portions shown in the drawings are not drawn in an actual proportional relationship for the convenience of description.
Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.
In all examples shown and discussed herein, any particular value should be construed as merely illustrative, and not limiting. Thus, other examples of the exemplary embodiments may have different values.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.
FIG. 1 is a flow diagram of a data processing method according to some embodiments of the invention. As shown in fig. 1, the data processing method of this embodiment includes steps S102 to S108.
In step S102, a data processing pressure value of the data reception point is acquired.
In some embodiments, the data received by the data receiving point is streaming data. The data sink may be, for example, a stream data processing platform in the big data processing system Spark, Storm.
The data processing pressure value may reflect the network pressure condition and the pressure condition of the data receiving point. The data volume in the network reflects the network pressure condition, and the performance resources of the data receiving point device reflect the device pressure condition.
In some embodiments, a network pressure indicator value and a data sink pressure indicator value may be collected, and a data processing pressure value determined from the network pressure indicator value and the data sink pressure indicator value. The network pressure indicator may include, for example, at least one of a flow rate and a flow rate of the network. The data sink pressure index may include at least one of a Central Processing Unit (CPU) utilization rate, a memory utilization rate, and a data processing time. Other indicators may be selected by those skilled in the art to calculate the data processing pressure value, as desired.
In some embodiments, the data processing pressure value may be determined based on a weighted value of the plurality of data processing pressure indicator values. The weights of the different types of data processing pressure indicators can be preset as required.
In step S104, the data processing pressure value is compared with a preset threshold value. The predetermined threshold may be a single value or a range of values.
In step S106, the data reception interval of the data reception point is adjusted according to the comparison result. The adjusted data reception interval may then be transmitted to the data reception point.
In some embodiments, data reception interval values of a plurality of levels may be preset, and a preset threshold range corresponding to each level may be set. Therefore, the data receiving interval value can be adjusted to the grade corresponding to the preset threshold range to which the current data processing pressure value belongs.
In step S108, the data receiving point intercepts and processes the data in batches according to the adjusted data receiving interval. The manner of adjustment may be to increase or decrease the data reception interval.
In some embodiments, the data reception interval reception interface may be defined based on the HTTP protocol or the Socket interface, and the adjustment script is preset at the data reception point. And when the data receiving point receives the adjusted data receiving interval, the numerical value is transmitted into the script so that the script can set or modify the data receiving interval.
By the method of the embodiment, the data receiving interval, namely the time window value, of the data receiving point can be dynamically adjusted according to the data processing pressure, so that the set data receiving interval is more matched with the actual requirement, and the stability of data processing and the resource utilization rate are improved.
An embodiment of the data reception interval adjustment method of the present invention is described below with reference to fig. 2.
Fig. 2 is a flow chart illustrating a method for adjusting a data reception interval according to some embodiments of the invention. As shown in fig. 2, the data reception interval adjusting method of this embodiment includes steps S202 to S206.
In step S202, the comparison result of the data processing pressure value with the preset threshold is determined. If the threshold value is larger than the preset high threshold value, executing the step S204; if the threshold value is smaller than the preset low threshold value, step S206 is executed.
In step S204, the data reception interval is narrowed.
In step S206, the data reception interval is increased.
The inventor analyzes the related technology and finds that when the data receiving interval is too large, the data processing of the same time batch is difficult to complete in time, and data accumulation is caused, so that the instability of the system is increased, and the delay of the data processing is increased; when the data receiving interval is too small, the data processing resources are idle, so that the resource utilization rate is low, and hardware resources and transmission resources are wasted.
Therefore, when the data processing pressure is higher, the data receiving interval can be reduced, so that the data volume processed in batches at the same time is reduced, and the stability of the system is improved; when the data pressure is small, the data receiving interval can be increased, so that data processing resources are fully utilized, the resource utilization rate of a cluster is improved, and the waste of hardware resources and the occupation of transmission resources are reduced.
In some embodiments, since the data volume and the network condition may fluctuate, the data processing pressure value within the preset time period may be monitored and then adjusted. An embodiment of the data reception interval adjustment method of the present invention is described below with reference to fig. 3.
Fig. 3 is a flowchart illustrating a data reception interval adjustment method according to another embodiment of the present invention. As shown in fig. 3, the data reception interval adjustment method of this embodiment includes steps S302 to S308.
In step S302, a data processing pressure value within a preset time period is calculated. For example, data processing pressure values corresponding to a plurality of time points within a preset time length can be calculated.
In step S304, the comparison result between the data processing pressure value and the preset threshold value within the preset time length is determined. If the threshold value is larger than the preset high threshold value, executing the step S306; if the threshold value is smaller than the preset low threshold value, step S308 is executed.
In step S306, the data reception interval is narrowed.
In step S308, the data reception interval is increased.
Therefore, the data receiving interval can be adjusted only under the condition that the data processing pressure is continuously larger or continuously smaller, so that the frequent adjustment of the data receiving interval is avoided, the system resource is saved, and the data processing efficiency is improved.
In some embodiments, the user may customize the monitoring parameters according to the needs, such as which metrics to monitor, when to monitor, and so forth. For example, a user selected data processing pressure indicator type and monitoring frequency may be received. Thus, the data processing pressure value of the data receiving point can be calculated according to the data processing pressure index type selected by the user. For the monitoring frequency, if the user selects real-time monitoring, the data processing pressure value is acquired and calculated in real time, and once the network fluctuates, the data receiving interval is adjusted; and if the user selects the timing monitoring, acquiring and calculating the data processing pressure value at regular time and adjusting. Therefore, the user can flexibly monitor and adjust the position according to the requirement.
An application scenario of an embodiment of the present invention is exemplarily described below. In this embodiment, an operator needs to collect signaling data of each phone call of a user, wireless port base station switching data when the user holds a mobile phone to move, Deep Packet Inspection (DPI) data when the user accesses the internet, and the like. These data serve as raw input data for the calculation of the user's position. The streaming data processing platform needs to receive these data from multiple systems and interfaces for data processing to obtain "user-location" information.
For example, when the user is idle around 12 o' clock in the middle of the night, the monitoring and calculation result shows that the data amount is small and the data processing pressure is small, so that the time window value (namely the data processing interval) can be increased; when the system is busy at about 9 am, the monitoring and calculation result shows that the data volume is large and the data processing pressure is large, so that the time window value can be reduced, and the received data can be processed in unit time.
An embodiment of the data processing device of the invention is described below with reference to fig. 4.
FIG. 4 is a block diagram of a data processing apparatus according to some embodiments of the invention. As shown in fig. 4, the data processing apparatus 400 of this embodiment includes: an acquisition module 4100 configured to acquire a data processing pressure value of the data receiving point; a comparison module 4200 configured to compare the data processing pressure value with a preset threshold; and an adjusting module 4300 configured to adjust the data receiving interval of the data receiving point according to the comparison result, so that the data receiving point intercepts and processes the data in batches according to the adjusted data receiving interval.
In some embodiments, the adjustment module 4300 is further configured to narrow the data reception interval if the data processing pressure value is greater than a preset high threshold; and under the condition that the data processing pressure value is smaller than the preset low threshold value, increasing the data receiving interval.
In some embodiments, the adjustment module 4300 is further configured to narrow the data reception interval if the data processing pressure value is greater than the preset high threshold for the preset duration; and under the condition that the data processing pressure values are all smaller than a preset low threshold value within the continuous preset time, increasing the data receiving interval.
In some embodiments, the data processing apparatus 400 further comprises: an acquisition module 4400 configured to acquire a network pressure index value and a data receiving point pressure index value; the obtaining module 4100 is further configured to determine a data processing pressure value according to the collected network pressure index value and the data receiving point pressure index value.
In some embodiments, the obtaining module 4100 is further configured to determine a data processing pressure value based on a weighted value of the plurality of data processing pressure indicator values.
An embodiment of the data processing system of the present invention is described below with reference to FIG. 5.
FIG. 5 is a block diagram of a data processing system according to some embodiments of the present invention. As shown in fig. 5, the data processing system 50 of this embodiment includes a data processing device 510 and a data receiving point 520. The data processing apparatus 400 in the embodiment of fig. 4 can be referred to for specific implementation of the data processing apparatus 510, and details are not repeated here. The data reception points 520 are configured to intercept and process data in batches according to the data reception intervals adjusted by the data processing device 510.
FIG. 6 is a block diagram of a data processing apparatus according to further embodiments of the present invention. As shown in fig. 6, the data processing apparatus 60 of this embodiment includes: a memory 610 and a processor 620 coupled to the memory 610, the processor 620 being configured to execute the data processing method of any of the foregoing embodiments based on instructions stored in the memory 610.
FIG. 7 is a block diagram of a data processing apparatus according to further embodiments of the present invention. As shown in fig. 7, the data processing apparatus 70 of this embodiment includes: the memory 710 and the processor 720 may further include an input/output interface 730, a network interface 740, a storage interface 750, and the like. These interfaces 730, 740, 750, as well as the memory 710 and the processor 720, may be connected, for example, by a bus 760. The input/output interface 730 provides a connection interface for input/output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 740 provides a connection interface for various networking devices. The storage interface 750 provides a connection interface for external storage devices such as an SD card and a usb disk.
An embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, wherein the program is configured to implement any one of the aforementioned data processing methods when executed by a processor.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (15)
1. A method of data processing, comprising:
acquiring a data processing pressure value of a data receiving point;
comparing the data processing pressure value with a preset threshold value;
and adjusting the data receiving interval of the data receiving point according to the comparison result so that the data receiving point intercepts and processes data in batches according to the adjusted data receiving interval.
2. The data processing method according to claim 1,
reducing a data receiving interval under the condition that the data processing pressure value is greater than a preset high threshold value;
and under the condition that the data processing pressure value is smaller than a preset low threshold value, increasing a data receiving interval.
3. The data processing method according to claim 1,
under the condition that the data processing pressure values are all larger than a preset high threshold value within a continuous preset time length, reducing a data receiving interval;
and under the condition that the data processing pressure values are all smaller than a preset low threshold value within a continuous preset time, increasing the data receiving interval.
4. The data processing method of claim 1, further comprising:
and acquiring a network pressure index value and a data receiving point pressure index value so as to determine a data processing pressure value according to the acquired network pressure index value and the acquired data receiving point pressure index value.
5. The data processing method of claim 4, wherein the network pressure indicator comprises at least one of flow and flow rate of the network, and the data receiving point pressure indicator comprises at least one of central processor utilization, memory utilization, and data processing time.
6. The data processing method of claim 1, wherein the obtaining a data processing pressure value for a data receiving point comprises:
and determining the data processing pressure value according to the weighted values of the various data processing pressure index values.
7. The data processing method according to any one of claims 1 to 6, wherein the data received by the data receiving point is streaming data.
8. A data processing apparatus comprising:
the acquisition module is configured to acquire a data processing pressure value of the data receiving point;
a comparison module configured to compare the data processing pressure value with a preset threshold;
and the adjusting module is configured to adjust the data receiving interval of the data receiving point according to the comparison result, so that the data receiving point intercepts and processes data in batches according to the adjusted data receiving interval.
9. The data processing apparatus according to claim 8, wherein the adjustment module is further configured to narrow a data reception interval if the data processing pressure value is greater than a preset high threshold; and under the condition that the data processing pressure value is smaller than a preset low threshold value, increasing a data receiving interval.
10. The data processing apparatus of claim 8, wherein the adjustment module is further configured to narrow a data reception interval if the data processing pressure values are each greater than a preset high threshold value for a preset length of time; and under the condition that the data processing pressure values are all smaller than a preset low threshold value within a continuous preset time, increasing the data receiving interval.
11. The data processing apparatus according to claim 8,
further comprising: an acquisition module configured to acquire a network pressure index value and a data receiving point pressure index value;
the acquisition module is further configured to determine a data processing pressure value from the collected network pressure index value and the data receiving point pressure index value.
12. The data processing apparatus of claim 8, wherein the acquisition module is further configured to determine the data processing pressure value based on a weighted value of the plurality of data processing pressure indicator values.
13. A data processing system comprising:
the data processing apparatus of any of claims 8 to 12; and the combination of (a) and (b),
a data receiving point configured to intercept and process data in batches according to the data receiving interval adjusted by the data processing apparatus.
14. A data processing apparatus comprising:
a memory; and
a processor coupled to the memory, the processor configured to perform the data processing method of any of claims 1-7 based on instructions stored in the memory.
15. A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, implements the data processing method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810931533.XA CN110838888A (en) | 2018-08-16 | 2018-08-16 | Data processing method, device, system and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810931533.XA CN110838888A (en) | 2018-08-16 | 2018-08-16 | Data processing method, device, system and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110838888A true CN110838888A (en) | 2020-02-25 |
Family
ID=69574172
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810931533.XA Pending CN110838888A (en) | 2018-08-16 | 2018-08-16 | Data processing method, device, system and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110838888A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070076726A1 (en) * | 2005-09-30 | 2007-04-05 | Microsoft Corporation | Receive window auto-tuning |
| CN1992582A (en) * | 2005-12-31 | 2007-07-04 | 中兴通讯股份有限公司 | Implementation method for adaptive variable slide receiving window of broadband signaling link |
| US20170078913A1 (en) * | 2015-09-14 | 2017-03-16 | Samsung Electronics Co., Ltd. | Electronic device and method for controlling transmission control protocol thereof |
-
2018
- 2018-08-16 CN CN201810931533.XA patent/CN110838888A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070076726A1 (en) * | 2005-09-30 | 2007-04-05 | Microsoft Corporation | Receive window auto-tuning |
| CN1992582A (en) * | 2005-12-31 | 2007-07-04 | 中兴通讯股份有限公司 | Implementation method for adaptive variable slide receiving window of broadband signaling link |
| US20170078913A1 (en) * | 2015-09-14 | 2017-03-16 | Samsung Electronics Co., Ltd. | Electronic device and method for controlling transmission control protocol thereof |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109918205B (en) | Edge equipment scheduling method, system, device and computer storage medium | |
| CN109981744B (en) | Data distribution method and device, storage medium and electronic equipment | |
| CN110247816B (en) | Index monitoring method and device | |
| CN109685092B (en) | Clustering method, equipment, storage medium and device based on big data | |
| CN109392002B (en) | Method and equipment for reporting network performance parameters | |
| CN109857528B (en) | Data migration speed adjusting method and device, storage medium and mobile terminal | |
| US20180253313A1 (en) | Operation instruction response control method and terminal for human-machine interface | |
| CN111682975B (en) | Network state prediction method, device, electronic equipment and storage medium | |
| CN107704317B (en) | Intelligent device and application management method thereof and device with storage function | |
| CN110247815B (en) | Data transmission method, device and system and computer readable storage medium | |
| CN111538572A (en) | Task processing method, device, scheduling server and medium | |
| JP4411296B2 (en) | Request restriction device, server device, request restriction method, request restriction program | |
| CN104462116B (en) | Data selection method and device | |
| CN112596985B (en) | IT asset detection method, device, equipment and medium | |
| CN114175748B (en) | Communication frequency adjusting method, device, electronic equipment and storage medium | |
| CN109885384B (en) | Task parallelism optimization method and device, computer equipment and storage medium | |
| CN113115327A (en) | Method, device, equipment and storage medium for dynamically optimizing network performance | |
| CN112468573A (en) | Data pushing method, device, equipment and storage medium | |
| CN110838888A (en) | Data processing method, device, system and computer readable storage medium | |
| CN108804152B (en) | Method and device for adjusting configuration parameters | |
| CN111124669A (en) | Operation method, system, terminal and storage medium of distributed SaaS software | |
| CN113873003B (en) | Data transmission control method and system between storage end and communication end in Internet of things | |
| CN107182067B (en) | Network optimization method and device | |
| CN107347196B (en) | Method and device for determining cell congestion | |
| CN114879832A (en) | Power consumption control method, device, chip, apparatus, and medium for arithmetic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200225 |
|
| RJ01 | Rejection of invention patent application after publication |