CN113760630A - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN113760630A CN113760630A CN202010564944.7A CN202010564944A CN113760630A CN 113760630 A CN113760630 A CN 113760630A CN 202010564944 A CN202010564944 A CN 202010564944A CN 113760630 A CN113760630 A CN 113760630A
- Authority
- CN
- China
- Prior art keywords
- data
- field
- length
- data set
- variable
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 38
- 230000005540 biological transmission Effects 0.000 claims abstract description 89
- 238000012545 processing Methods 0.000 claims abstract description 67
- 238000000034 method Methods 0.000 claims abstract description 39
- 238000004140 cleaning Methods 0.000 claims description 27
- 238000004590 computer program Methods 0.000 claims description 9
- 238000004364 calculation method Methods 0.000 abstract description 55
- 230000008569 process Effects 0.000 abstract description 23
- 238000010586 diagram Methods 0.000 description 17
- 238000004891 communication Methods 0.000 description 7
- 230000000694 effects Effects 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 238000007667 floating Methods 0.000 description 5
- 238000012544 monitoring process Methods 0.000 description 5
- 238000011161 development Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000000750 progressive effect Effects 0.000 description 3
- 230000001960 triggered effect Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000005406 washing Methods 0.000 description 2
- 241000508269 Psidium Species 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3485—Performance evaluation by tracing or monitoring for I/O devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5022—Mechanisms to release resources
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a data processing method and device, and relates to the technical field of computers. One embodiment of the method comprises: acquiring target data in real time, and carrying out serialization processing on the target data; placing the target data subjected to serialization processing in a data set, and determining the length of a field corresponding to the data set; judging whether the length of a field corresponding to the data set is less than or equal to a data transmission threshold value; if so, executing the caching operation on the data set, and transmitting the data set executing the caching operation according to the transmission protocol. The implementation mode improves the calculation efficiency in the data processing process, reduces the calculation amount, reduces the system performance loss, releases the memory space and saves the system resources.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data processing method and apparatus.
Background
With the high-speed development of the information-oriented era, the requirements of people on the speed and the quantity of acquired information are higher and higher, but the system load is easily overlarge due to the fact that a large amount of data are transmitted, and risks such as downtime are easily caused. In order to solve the above problems, a current limiting tool such as ratelimit (applied to guava (a java library based on open source)) which restricts the maximum number of accesses in a period of time is often used in the existing method.
In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art:
in the existing method, accurate calculation needs to be carried out after the transmission data packet is assembled, the calculation efficiency is low, the calculation amount is large, and the system performance loss is increased; and when the data packet to be transmitted is too large, the memory space is occupied too much, so that a large amount of system resources are occupied.
Disclosure of Invention
In view of this, embodiments of the present invention provide a data processing method and apparatus, which can improve the calculation efficiency in the data processing process, reduce the calculation amount, reduce the system performance loss, release the memory space, and save the system resources.
To achieve the above object, according to a first aspect of embodiments of the present invention, there is provided a data processing method including:
acquiring target data in real time, and carrying out serialization processing on the target data;
placing the target data subjected to serialization processing in a data set, and determining the length of a field corresponding to the data set;
judging whether the length of a field corresponding to the data set is less than or equal to a data transmission threshold value;
if so, executing the caching operation on the data set, and transmitting the data set executing the caching operation according to the transmission protocol.
Further, the step of placing the target data after the serialization processing in the data set and determining the field length corresponding to the data set includes: and placing the target data subjected to serialization processing in a data set, dividing the target data into fixed-length field data and variable-length field data according to the data type, and determining the field length corresponding to the data set according to the field lengths corresponding to the fixed-length field data and the variable-length field data.
Further, the step of determining the field length corresponding to the data set according to the field lengths corresponding to the fixed-length field data and the variable-length field data further includes: and determining the field length accuracy according to the field length of the fixed-length field data and the field length of the variable-length field data, and taking the field length corresponding to the variable-length field data as the field length corresponding to the data set under the condition that the field length accuracy is greater than or equal to an accuracy threshold.
Further, before the step of determining whether the field length of the data set is less than or equal to the data transmission threshold, the data processing method further includes: and performing data cleaning on the target data in the data set according to the data attribute corresponding to the target data and the field length threshold, wherein the data attribute indicates the service type corresponding to the target data.
Further, the data processing method further comprises: and judging whether the target data subjected to data cleaning is variable-length field data, if so, adjusting the field length of the data set after the data cleaning.
Further, in the case that the field length of the data set is greater than the data transmission threshold, the data processing method further includes: and triggering a current limiting rule, and stopping executing the caching operation on the data set.
According to a second aspect of the embodiments of the present invention, there is provided a data processing apparatus including:
the target data acquisition module is used for acquiring target data in real time and carrying out serialization processing on the target data;
the field length determining module is used for placing the target data subjected to serialization processing in a data set and determining the field length corresponding to the data set;
the judging module is used for judging whether the field length corresponding to the data set exceeds a data transmission threshold value;
and the processing module is used for executing cache operation on the data set under the condition that the length of the field corresponding to the data set is less than or equal to the data transmission threshold value, and transmitting the data set executing the cache operation according to the transmission protocol.
Further, the field length determination module is further configured to: dividing target data into fixed-length field data and variable-length field data according to the data types, determining field length accuracy according to the field length of the fixed-length field data and the field length of the variable-length field data, and taking the field length corresponding to the variable-length field data as the field length corresponding to the data set under the condition that the field length accuracy is greater than or equal to an accuracy threshold.
According to a third aspect of embodiments of the present invention, there is provided an electronic apparatus, including:
one or more processors;
a storage device for storing one or more programs,
when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement any of the data processing methods described above.
According to a fourth aspect of embodiments of the present invention, there is provided a computer-readable medium on which a computer program is stored, the program, when executed by a processor, implementing any one of the data processing methods described above.
One embodiment of the above invention has the following advantages or benefits: the target data is acquired in real time, and the target data is serialized; placing the target data subjected to serialization processing in a data set, and determining the length of a field corresponding to the data set; judging whether the length of a field corresponding to the data set is less than or equal to a data transmission threshold value; if so, performing caching operation on the data set, and transmitting the data set subjected to the caching operation according to the transmission protocol, so that the technical means that the data set subjected to the caching operation is accurately calculated after the transmission data packet is assembled in the prior art is overcome, the calculation efficiency is low, the calculation amount is large, and the system performance loss is increased; and when the data packet to be transmitted is too large, the memory space is occupied too much, which causes the technical problem that a large amount of system resources are occupied, thereby achieving the technical effects of improving the calculation efficiency in the data processing process, reducing the calculation amount, reducing the system performance loss, releasing the memory space and saving the system resources.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
fig. 1 is a schematic diagram of a main flow of a data processing method provided according to a first embodiment of the present invention;
FIG. 2a is a schematic diagram of a main flow of a data processing method according to a second embodiment of the present invention;
FIG. 2b is a schematic diagram of a current limiter corresponding to the data processing method of FIG. 2a used in an application program;
FIG. 2c is a schematic diagram of the main modules of a data processing apparatus corresponding to the data processing method shown in FIG. 2 a;
FIG. 2d is a schematic flow chart illustrating a process of acquiring target data and performing serialization on the target data in the data processing method shown in FIG. 2 a;
FIG. 2e is a diagram illustrating a division of target data into fixed-length field data and variable-length field data in the data processing method of FIG. 2 a;
FIG. 2f is a schematic diagram of a data packet provided by the data processing method of FIG. 2 a;
FIG. 3 is a schematic diagram of the main blocks of a data processing apparatus according to an embodiment of the present invention;
FIG. 4 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
fig. 5 is a schematic block diagram of a computer system suitable for use in implementing a terminal device or server of an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a schematic diagram of a main flow of a data processing method provided according to a first embodiment of the present invention; as shown in fig. 1, the data processing method provided in the embodiment of the present invention mainly includes:
and step S101, acquiring target data in real time and carrying out serialization processing on the target data.
Specifically, according to the embodiment of the present invention, by acquiring target data (such as cache object data and transmission object data) in real time, it is beneficial to determine the field length corresponding to the data set by using a progressive calculation method in the subsequent process of assembling the target data into the data set, so that the calculation efficiency of the data set is improved, and further, the efficiency of subsequent data transmission is improved.
Serialization: refers to the process of converting state information of an object into a form that can be stored or transmitted.
And S102, placing the target data subjected to serialization processing in a data set, and determining the field length corresponding to the data set.
According to the embodiment of the present invention, the step of placing the target data subjected to the serialization processing in the data set and determining the field length corresponding to the data set includes: and placing the target data subjected to serialization processing in a data set, dividing the target data into fixed-length field data and variable-length field data according to the data type, and determining the field length corresponding to the data set according to the field lengths corresponding to the fixed-length field data and the variable-length field data.
Specifically, in the software development process, many development languages provide rich data types, and target data can be divided into fixed-length field data (such as target data with data types of boolean, integer, long integer, floating point number, and the like) and variable-length field data (such as target data with data types of character strings) according to the number of occupied bytes corresponding to different data types.
Through the setting, the target data is divided into fixed-length field data and variable-length field data according to the data type, wherein the corresponding field length of the fixed-length field data is fixed, and the corresponding field length of the fixed-length field data can be determined according to the length of the byte occupied by the specific data type; for the variable length field, since the corresponding field length is not fixed, calculation needs to be performed according to the specific stored content. Through the arrangement, the method for calculating the field lengths of part of target data is simplified, namely the field lengths in the data set are estimated, so that the calculated amount before data transmission is reduced, and the calculation efficiency is improved.
Further, according to an embodiment of the present invention, the step of determining the field length corresponding to the data set according to the field lengths corresponding to the fixed length field data and the variable length field data further includes: and determining the field length accuracy according to the field length of the fixed-length field data and the field length of the variable-length field data, and taking the field length corresponding to the variable-length field data as the field length corresponding to the data set under the condition that the field length accuracy is greater than or equal to an accuracy threshold.
In practical application, the current limiting operation triggered and executed in many service scenarios is performed within a certain range, so that the field length of the data to be transmitted (i.e. the data set containing the target data) does not need to be accurately calculated. Through the arrangement, only the field length corresponding to the variable-length field data is calculated according to the field length accuracy rate of the service scene corresponding to the execution of the current limiting operation and is used as the field length corresponding to the data set, so that the calculation amount of the assembled data packet (namely the data set) before data transmission is further reduced, the calculation efficiency is improved, and the system performance loss is reduced.
Current limiting operation: the data transmission method is characterized in that when the data volume is too large in the data transmission process, great pressure is applied to data bandwidth and system performance, the risk of system downtime exists, and the operation of limiting the data transmission flow is adopted for reducing the system pressure.
According to a specific implementation manner of the embodiment of the present invention, the field length accuracy may be determined by using a ratio of a field length of the variable length field data to a total field length of the variable length field data and the fixed length field data.
It should be noted that the field length accuracy is set by utilizing that in most cases, the field length corresponding to the variable length field data is much larger than the field length corresponding to the fixed length field data. If yes, in a certain data set, the field length corresponding to the fixed-length field data is far larger than that of the variable-length field data, and the determination mode of the field length accuracy rate is also adjusted adaptively.
Step S103, judging whether the length of the field corresponding to the data set is less than or equal to the data transmission threshold value.
In the above setting, the data transmission threshold may be determined according to actual conditions (such as bandwidth, signal strength, server resources, etc.).
Specifically, according to the embodiment of the present invention, before the step of determining whether the field length of the data set is less than or equal to the data transmission threshold, the data processing method further includes: and performing data cleaning on the target data in the data set according to the data attribute corresponding to the target data and the field length threshold, wherein the data attribute indicates the service type corresponding to the target data.
For example, for data transmission of the monitoring service, the acquired data of the system operation log, the server resource, and the like of the service system is non-service data that is not related to the monitoring service, and the priority and the importance are also relatively low, so that the target data for the data attribute may be discarded (i.e., the above-mentioned data washing is performed). According to a specific implementation manner of the embodiment of the present invention, for a certain data packet (e.g., a certain subset of a data set), the field length corresponding to the certain data packet exceeds a field length threshold, that is, the data packet is very large, and not only the memory space of a service system needs to be occupied, but also a large network overhead is occupied in a subsequent data transmission process, and at the same time, a risk that the field length corresponding to the data set exceeds a data transmission threshold is caused, the target data may also be subjected to data cleaning, so as to release the memory space and save system resources.
Further, according to the embodiment of the present invention, the data processing method further includes; and judging whether the target data subjected to data cleaning is variable-length field data, if so, adjusting the field length of the data set after the data cleaning.
After the target data in the data set is subjected to data cleaning, the field length corresponding to the data set changes correspondingly, but the triggering and executing of the current limiting operation in most of the service scenes is performed within a certain range, and the field length of the data to be transmitted (namely, the data set containing the target data) does not need to be accurately calculated, so that the field length corresponding to the data set can be estimated by calculating the field length corresponding to the variable-length field data, the calculation amount is reduced, and the calculation efficiency is improved. By the arrangement, the field length corresponding to the data set can be changed certainly after the data is cleaned, and whether the field length corresponding to the data set needs to be recalculated can be determined by judging whether the object of the data cleaning is the variable-length field data. When the target data for data cleaning is fixed-length field data, the field length corresponding to the data set can be adjusted, and the method for estimating the field length corresponding to the data set by the field length corresponding to the variable-length field data is continuously adopted, so that the calculation amount is further reduced, the calculation efficiency is improved, and the system performance loss is reduced.
And step S104, sending the data set to a cache device under the condition that the length of the field corresponding to the data set is less than or equal to the data transmission threshold value, and transmitting the data set according to the transmission protocol.
Through the arrangement, the data set is sent to the cache device, so that the subsequent data transmission efficiency can be improved, the data transmission threshold value can be flexibly set, the size of the data set is limited, and the network overhead and bandwidth pressure for transmitting the data set are reduced.
Further, according to the embodiment of the present invention, in the case that the field length of the data set is greater than the data transmission threshold, the data processing method further includes: and triggering a current limiting rule, and stopping executing the caching operation on the data set.
And (3) current limiting rule: and if the field length corresponding to the data to be transmitted (namely the data set) exceeds the data transmission threshold, pausing the assembly operation of the corresponding data to be transmitted and stopping sending the data to the cache device.
According to the technical scheme of the embodiment of the invention, the target data is acquired in real time and is subjected to serialization processing; placing the target data subjected to serialization processing in a data set, and determining the length of a field corresponding to the data set; judging whether the length of a field corresponding to the data set is less than or equal to a data transmission threshold value; if so, performing caching operation on the data set, and transmitting the data set subjected to the caching operation according to the transmission protocol, so that the technical means that the data set subjected to the caching operation is accurately calculated after the transmission data packet is assembled in the prior art is overcome, the calculation efficiency is low, the calculation amount is large, and the system performance loss is increased; and when the data packet to be transmitted is too large, the memory space is occupied too much, which causes the technical problem that a large amount of system resources are occupied, thereby achieving the technical effects of improving the calculation efficiency in the data processing process, reducing the calculation amount, reducing the system performance loss, releasing the memory space and saving the system resources.
FIG. 2a is a schematic diagram of a main flow of a data processing method according to a second embodiment of the present invention; the embodiment of the present invention is an application scenario for processing data of an application program, and as shown in fig. 2b and 2c, a data processing device corresponding to the data processing method provided by the embodiment of the present invention may be disposed in a current limiter, and the current limiter executes the data processing method provided by the embodiment of the present invention, and the current limiter may be disposed inside the application program; as shown in fig. 2a, the data processing method provided in the embodiment of the present invention mainly includes:
step S201, acquiring target data in real time, and performing serialization processing on the target data.
Specifically, according to the embodiment of the present invention, by acquiring target data such as cache object data and transmission object data in the application in real time, it is beneficial to determine the field length corresponding to the data set by using a progressive calculation method in the subsequent process of assembling the target data into the data set, so that the calculation efficiency of the data set is improved, and further, the efficiency of subsequent data transmission is improved.
Specifically, according to the embodiment of the present invention, the serialization refers to a process of converting the state information of the object into a form that can be stored or transmitted. Specifically, as shown in fig. 2d, the process corresponding to step S201 acquires target data (i.e., an object to be transmitted in the graph), and performs serialization processing on the target data (i.e., a process of converting state information of the object into a form capable of being stored or transmitted), so that the target data is converted into a form capable of being transmitted over a network.
Step S202, the target data after the serialization processing is arranged in a data set, and the target data is divided into fixed length field data and variable length field data according to the data type.
Specifically, in the software development process, many development languages provide rich data types, and target data can be divided into fixed-length field data (such as target data with data types of boolean, integer, long integer, floating point number, and the like) and variable-length field data (such as target data with data types of character strings) according to the number of occupied bytes corresponding to different data types.
As shown in fig. 2e, for java (an object-oriented programming language), it provides that boolean (boolean) data occupies 1 byte, floating point (float) data occupies 4 bytes, double precision floating point (double) occupies 8 bytes, integer (int) data occupies 4 bytes, and long integer (long) data occupies 8 bytes, so that it can be divided into fixed-length field data a1(ii) a The bytes occupied by the string are not fixed, the specific length is determined by the content stored in the string, and the string can be divided into variable-length field data x.
Step S203, determining the field length accuracy rate according to the field length of the fixed-length field data and the field length of the variable-length field data.
Through the setting, the target data is divided into fixed-length field data and variable-length field data according to the data type, wherein the corresponding field length of the fixed-length field data is fixed, and the corresponding field length of the fixed-length field data can be determined according to the length of the byte occupied by the specific data type; for the variable length field, since the corresponding field length is not fixed, calculation needs to be performed according to the specific stored content. Through the arrangement, the method for calculating the field lengths of part of target data is simplified, namely the field lengths in the data set are estimated, so that the calculated amount before data transmission is reduced, and the calculation efficiency is improved.
And step S204, under the condition that the field length accuracy is greater than or equal to the accuracy threshold, taking the field length corresponding to the variable-length field data as the field length corresponding to the data set.
In practical application, the current limiting operation triggered and executed in many service scenarios is performed within a certain range, so that the field length of the data to be transmitted (i.e. the data set containing the target data) does not need to be accurately calculated. Through the arrangement, only the field length corresponding to the variable-length field data is calculated according to the field length accuracy rate of the service scene corresponding to the execution of the current limiting operation and is used as the field length corresponding to the data set, so that the calculation amount of the assembled data packet (namely the data set) before data transmission is further reduced, the calculation efficiency is improved, and the system performance loss is reduced.
According to the embodiment of the present invention, the theoretical basis of the step S204 is:
the field length f (x) corresponding to the data set is the field length f (y) of the serialized target data + the field length a added by the network protocol0；
Field length f (y) of serialized target data, i.e. field length x corresponding to variable-length field data and field length a corresponding to fixed-length field1。
The following relationship can be obtained by substituting the equation:
f(x)＝x+a0+a1 (1)
due to a0And a1Are all constants, so let:
b＝a0+a1 (2)
combining formula (1) and formula (2) gives:
f(x)＝x+b (3)
as can be seen from equation 3, the field length corresponding to the data set is ultimately mainly determined by the field length x corresponding to the variable length field.
When x is much larger than b, the effect of b can be neglected, i.e.
f(x)≈x (4)
From the equation (4), when the field length accuracy is greater than or equal to the accuracy threshold (i.e., when the field length corresponding to the variable-length field data is greater than the field length corresponding to the fixed-length field data), the field length corresponding to the variable-length field can be used to estimate the field length corresponding to the data set.
And S205, performing data cleaning on the target data in the data set according to the data attribute and the field length threshold corresponding to the target data.
When the corresponding service of the application program needs to be monitored so as to ensure that the service system can normally operate, the feedback can be timely made under the condition that the service system has a fault, and a system maintenance worker is informed to process the fault. In order to complete the monitoring service, the operation logs of the service system, server resources and the like need to be collected as target data, but the data attributes of the target data belong to non-service data, the corresponding priority and importance of the target data are low, and in order not to influence the normal operation of the service system, the target data can be subjected to data cleaning. In another case, for a certain data packet (e.g., a certain subset of a data set), the corresponding field length exceeds the field length threshold (i.e., abnormal target data), that is, the data packet is very large, which not only needs to occupy the memory space of the service system, but also occupies a large network overhead in the subsequent data transmission process, and at the same time, causes a risk that the corresponding field length of the data set exceeds the data transmission threshold, and the target data may also be subjected to data cleaning to release the memory space and save system resources.
Step S206, judging whether the target data for data cleaning is variable-length field data, if so, adjusting the field length of the data set after the data cleaning.
Specifically, the size of the data packet (i.e. the field length corresponding to the above-mentioned data set) is composed of a variable length content (i.e. the field length corresponding to the variable length field data) x and a fixed length content (i.e. the field length a corresponding to the fixed length field data)1And field length a of transport protocol addend0) b, determining. In the java language, the data packet (i.e. the data set) collected by the service system is represented as shown in fig. 2 f.
Further, according to the embodiment of the present invention, the data processing method further includes; and judging whether the target data subjected to data cleaning is variable-length field data, if so, adjusting the field length of the data set after the data cleaning.
By the arrangement, the field length corresponding to the data set can be changed certainly after the data is cleaned, and whether the field length corresponding to the data set needs to be recalculated can be determined by judging whether the object of the data cleaning is the variable-length field data. When the target data for data cleaning is fixed-length field data, the field length corresponding to the data set can be adjusted, and the method for estimating the field length corresponding to the data set by the field length corresponding to the variable-length field data is continuously adopted, so that the calculation amount is further reduced, the calculation efficiency is improved, and the system performance loss is reduced.
In fig. 2f, the fixed-length content is composed of a field 3, a start time, and an offset time, and the field length corresponding to the fixed-length content is:
b＝int(4)+long(8)+long(8)＝20
the field length of the data set is defined as size, and the data transmission threshold is defined as m. Assuming that m is 1MB, 1024, 1048576 bytes, since the length is much larger than the field length 20 corresponding to the fixed-length content, the influence of the fixed-length content can be ignored at this time, and the accuracy of the field after being ignored is: 1024 × 1024/(1024 × 1024+20) ≈ 99.997%, and meets the requirement of an accuracy threshold.
Step S207, determining whether the field length corresponding to the data set is less than or equal to the data transmission threshold. If yes, that is, the length of the field corresponding to the data set is less than or equal to the data transmission threshold, then step S208 is executed; if not, that is, the field length corresponding to the data set is greater than the data transmission threshold, step S209 is executed.
In the above setting, the data transmission threshold may be determined according to actual conditions.
And step S208, performing caching operation on the data set, and transmitting the data set subjected to the caching operation according to the transmission protocol.
Through the arrangement, the data set is sent to the cache device (namely, the cache processing is executed), so that the efficiency of subsequent data transmission can be improved, the data transmission threshold value can be flexibly set, the size of the data set is limited, and the network overhead and the bandwidth pressure of the transmission processing of the data set are reduced.
Step S209, triggering a current limiting rule, and stopping performing a caching operation on the data set.
And (3) current limiting rule: and if the field length corresponding to the data to be transmitted (namely the data set) exceeds the data transmission threshold, pausing the assembly operation of the corresponding data to be transmitted and stopping sending the data to the cache device.
According to the technical scheme of the embodiment of the invention, the target data is acquired in real time and is subjected to serialization processing; placing the target data subjected to serialization processing in a data set, and determining the length of a field corresponding to the data set; judging whether the length of a field corresponding to the data set is less than or equal to a data transmission threshold value; if so, performing caching operation on the data set, and transmitting the data set subjected to the caching operation according to the transmission protocol, so that the technical means that the data set subjected to the caching operation is accurately calculated after the transmission data packet is assembled in the prior art is overcome, the calculation efficiency is low, the calculation amount is large, and the system performance loss is increased; and when the data packet to be transmitted is too large, the memory space is occupied too much, which causes the technical problem that a large amount of system resources are occupied, thereby achieving the technical effects of improving the calculation efficiency in the data processing process, reducing the calculation amount, reducing the system performance loss, releasing the memory space and saving the system resources.
FIG. 3 is a schematic diagram of the main blocks of a data processing apparatus according to an embodiment of the present invention; as shown in fig. 3, a data processing apparatus 300 according to an embodiment of the present invention mainly includes:
and the target data acquisition module 301 is configured to acquire target data in real time and perform serialization processing on the target data.
Specifically, according to the embodiment of the present invention, by acquiring target data (such as cache object data and transmission object data) in real time, it is beneficial to determine the field length corresponding to the data set by using a progressive calculation method in the subsequent process of assembling the target data into the data set, so that the calculation efficiency of the data set is improved, and further, the efficiency of subsequent data transmission is improved.
Serialization: refers to the process of converting state information of an object into a form that can be stored or transmitted.
A field length determining module 302, configured to place the serialized target data in a data set, and determine a field length corresponding to the data set.
According to an embodiment of the present invention, the field length determining module 302 is further configured to: and placing the target data subjected to serialization processing in a data set, dividing the target data into fixed-length field data and variable-length field data according to the data type, and determining the field length corresponding to the data set according to the field lengths corresponding to the fixed-length field data and the variable-length field data.
Specifically, in the software development process, many development languages provide rich data types, and target data can be divided into fixed-length field data (such as target data with data types of boolean, integer, long integer, floating point number, and the like) and variable-length field data (such as target data with data types of character strings) according to the number of occupied bytes corresponding to different data types.
Through the setting, the target data is divided into fixed-length field data and variable-length field data according to the data type, wherein the corresponding field length of the fixed-length field data is fixed, and the corresponding field length of the fixed-length field data can be determined according to the length of the byte occupied by the specific data type; for the variable length field, since the corresponding field length is not fixed, calculation needs to be performed according to the specific stored content. Through the arrangement, the method for calculating the field lengths of part of target data is simplified, namely the field lengths in the data set are estimated, so that the calculated amount before data transmission is reduced, and the calculation efficiency is improved.
Further, according to an embodiment of the present invention, the field length determining module 302 is further configured to: and determining the field length accuracy according to the field length of the fixed-length field data and the field length of the variable-length field data, and taking the field length corresponding to the variable-length field data as the field length corresponding to the data set under the condition that the field length accuracy is greater than or equal to an accuracy threshold.
In practical application, the current limiting operation triggered and executed in many service scenarios is performed within a certain range, so that the field length of the data to be transmitted (i.e. the data set containing the target data) does not need to be accurately calculated. Through the arrangement, only the field length corresponding to the variable-length field data is calculated according to the field length accuracy rate of the service scene corresponding to the execution of the current limiting operation and is used as the field length corresponding to the data set, so that the calculation amount of the assembled data packet (namely the data set) before data transmission is further reduced, the calculation efficiency is improved, and the system performance loss is reduced.
Current limiting operation: the data transmission method is characterized in that when the data volume is too large in the data transmission process, great pressure is applied to data bandwidth and system performance, the risk of system downtime exists, and the operation of limiting the data transmission flow is adopted for reducing the system pressure.
According to a specific implementation manner of the embodiment of the present invention, the field length accuracy may be determined by using a ratio of a field length of the variable length field data to a total field length of the variable length field data and the fixed length field data.
It should be noted that the field length accuracy is set by utilizing that in most cases, the field length corresponding to the variable length field data is much larger than the field length corresponding to the fixed length field data. If yes, in a certain data set, the field length corresponding to the fixed-length field data is far larger than that of the variable-length field data, and the determination mode of the field length accuracy rate is also adjusted adaptively.
The determining module 303 is configured to determine whether a field length corresponding to the data set exceeds a data transmission threshold.
In the above setting, the data transmission threshold may be determined according to actual conditions.
Specifically, according to the embodiment of the present invention, the data processing apparatus 300 further includes a data cleansing module, before the step of determining whether the field length of the data set is less than or equal to the data transmission threshold, the data cleansing module is configured to: and performing data cleaning on the target data in the data set according to the data attribute corresponding to the target data and the field length threshold, wherein the data attribute indicates the service type corresponding to the target data.
For example, for data transmission of the monitoring service, the acquired data of the system operation log, the server resource, and the like of the service system is non-service data that is not related to the monitoring service, and the priority and the importance are also relatively low, so that the target data for the data attribute may be discarded (i.e., the above-mentioned data washing is performed). According to a specific implementation manner of the embodiment of the present invention, for a certain data packet (e.g., a certain subset of a data set), the field length corresponding to the certain data packet exceeds a field length threshold, that is, the data packet is very large, and not only the memory space of a service system needs to be occupied, but also a large network overhead is occupied in a subsequent data transmission process, and at the same time, a risk that the field length corresponding to the data set exceeds a data transmission threshold is caused, the target data may also be subjected to data cleaning, so as to release the memory space and save system resources.
Further, according to the embodiment of the present invention, the data processing apparatus 300 further includes an adjusting module, configured to determine whether the target data for data cleansing is variable-length field data, and if so, adjust the field length of the data set after data cleansing.
After the target data in the data set is subjected to data cleaning, the field length corresponding to the data set changes correspondingly, but the triggering and executing of the current limiting operation in most of the service scenes is performed within a certain range, and the field length of the data to be transmitted (namely, the data set containing the target data) does not need to be accurately calculated, so that the field length corresponding to the data set can be estimated by calculating the field length corresponding to the variable-length field data, the calculation amount is reduced, and the calculation efficiency is improved. By the arrangement, the field length corresponding to the data set can be changed certainly after the data is cleaned, and whether the field length corresponding to the data set needs to be recalculated can be determined by judging whether the object of the data cleaning is the variable-length field data. When the target data for data cleaning is fixed-length field data, the field length corresponding to the data set can be adjusted, and the method for estimating the field length corresponding to the data set by the field length corresponding to the variable-length field data is continuously adopted, so that the calculation amount is further reduced, the calculation efficiency is improved, and the system performance loss is reduced.
And the processing module 304 is configured to send the data set to the cache device and perform transmission processing on the data set according to the transmission protocol under the condition that the length of the field corresponding to the data set is less than or equal to the data transmission threshold.
Through the arrangement, the data set is sent to the cache device, so that the subsequent data transmission efficiency can be improved, the data transmission threshold value can be flexibly set, the size of the data set is limited, and the network overhead and bandwidth pressure for transmitting the data set are reduced.
Further, according to the embodiment of the present invention, the data processing apparatus 300 further includes a current limiting rule triggering module, where the current limiting rule triggering module is configured to, when the field length of the data set is greater than the data transmission threshold: and triggering a current limiting rule, and stopping executing the caching operation on the data set.
And (3) current limiting rule: and if the field length corresponding to the data to be transmitted (namely the data set) exceeds the data transmission threshold, pausing the assembly operation of the corresponding data to be transmitted and stopping sending the data to the cache device.
According to the technical scheme of the embodiment of the invention, the target data is acquired in real time and is subjected to serialization processing; placing the target data subjected to serialization processing in a data set, and determining the length of a field corresponding to the data set; judging whether the length of a field corresponding to the data set is less than or equal to a data transmission threshold value; if so, performing caching operation on the data set, and transmitting the data set subjected to the caching operation according to the transmission protocol, so that the technical means that the data set subjected to the caching operation is accurately calculated after the transmission data packet is assembled in the prior art is overcome, the calculation efficiency is low, the calculation amount is large, and the system performance loss is increased; and when the data packet to be transmitted is too large, the memory space is occupied too much, which causes the technical problem that a large amount of system resources are occupied, thereby achieving the technical effects of improving the calculation efficiency in the data processing process, reducing the calculation amount, reducing the system performance loss, releasing the memory space and saving the system resources.
Fig. 4 shows an exemplary system architecture 400 of a data processing method or data processing apparatus to which embodiments of the present invention may be applied.
As shown in fig. 4, the system architecture 400 may include terminal devices 401, 402, 403, a network 404, and a server 405 (this architecture is merely an example, and the components included in a particular architecture may be adapted according to application specific circumstances). The network 404 serves as a medium for providing communication links between the terminal devices 401, 402, 403 and the server 405. Network 404 may include various types of connections, such as wire, wireless communication links, or fiber optic cables, to name a few.
A user may use terminal devices 401, 402, 403 to interact with a server 405 over a network 404 to receive or send messages or the like. The terminal devices 401, 402, 403 may have installed thereon various communication client applications, such as shopping-like applications, web browser applications, search-like applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only).
The terminal devices 401, 402, 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 405 may be a server providing various services, such as a background management server (for example only) providing support for shopping websites browsed by users using the terminal devices 401, 402, 403. The background management server may analyze and perform other processing on the received data such as the target data, and feed back a processing result (for example, a data set and a field length corresponding to the data set — only an example) to the terminal device.
It should be noted that the data processing method provided by the embodiment of the present invention is generally executed by the server 405, and accordingly, the data processing apparatus is generally disposed in the server 405.
It should be understood that the number of terminal devices, networks, and servers in fig. 4 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 5, shown is a block diagram of a computer system 500 suitable for use with a terminal device implementing an embodiment of the present invention. The terminal device shown in fig. 5 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 5, the computer system 500 includes a Central Processing Unit (CPU)501 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)502 or a program loaded from a storage section 508 into a Random Access Memory (RAM) 503. In the RAM 503, various programs and data necessary for the operation of the system 500 are also stored. The CPU 501, ROM 502, and RAM 503 are connected to each other via a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.
The following components are connected to the I/O interface 505: an input portion 506 including a keyboard, a mouse, and the like; an output portion 507 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 508 including a hard disk and the like; and a communication section 509 including a network interface card such as a LAN card, a modem, or the like. The communication section 509 performs communication processing via a network such as the internet. The driver 510 is also connected to the I/O interface 505 as necessary. A removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 510 as necessary, so that a computer program read out therefrom is mounted into the storage section 508 as necessary.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 509, and/or installed from the removable medium 511. The computer program performs the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 501.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor comprises a target data acquisition module, a field length determination module, a judgment module and a processing module. The names of these modules do not limit the modules themselves in some cases, for example, the target data acquiring module may also be described as a "module for acquiring target data in real time and performing serialization processing on the target data".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: acquiring target data in real time, and carrying out serialization processing on the target data; placing the target data subjected to serialization processing in a data set, and determining the length of a field corresponding to the data set; judging whether the length of a field corresponding to the data set is less than or equal to a data transmission threshold value; if so, executing the caching operation on the data set, and transmitting the data set executing the caching operation according to the transmission protocol.
According to the technical scheme of the embodiment of the invention, the target data is acquired in real time and is subjected to serialization processing; placing the target data subjected to serialization processing in a data set, and determining the length of a field corresponding to the data set; judging whether the length of a field corresponding to the data set is less than or equal to a data transmission threshold value; if so, performing caching operation on the data set, and transmitting the data set subjected to the caching operation according to the transmission protocol, so that the technical means that the data set subjected to the caching operation is accurately calculated after the transmission data packet is assembled in the prior art is overcome, the calculation efficiency is low, the calculation amount is large, and the system performance loss is increased; and when the data packet to be transmitted is too large, the memory space is occupied too much, which causes the technical problem that a large amount of system resources are occupied, thereby achieving the technical effects of improving the calculation efficiency in the data processing process, reducing the calculation amount, reducing the system performance loss, releasing the memory space and saving the system resources.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A data processing method, comprising:
acquiring target data in real time, and carrying out serialization processing on the target data;
placing the target data subjected to serialization processing in a data set, and determining the length of a field corresponding to the data set;
judging whether the length of the field corresponding to the data set is less than or equal to a data transmission threshold value;
if so, executing cache operation on the data set, and transmitting the data set executing the cache operation according to a transmission protocol.
2. The data processing method according to claim 1, wherein the step of placing the serialized target data into a data set, and the step of determining a field length corresponding to the data set comprises: the method comprises the steps of placing target data subjected to serialization processing in a data set, dividing the target data into fixed-length field data and variable-length field data according to data types, and determining field lengths corresponding to the data set according to field lengths corresponding to the fixed-length field data and the variable-length field data.
3. The data processing method according to claim 2, wherein the step of determining the field length corresponding to the data set according to the field lengths corresponding to the fixed-length field data and the variable-length field data further comprises: and determining a field length accuracy rate according to the field length of the fixed-length field data and the field length of the variable-length field data, and taking the field length corresponding to the variable-length field data as the field length corresponding to the data set under the condition that the field length accuracy rate is greater than or equal to an accuracy rate threshold.
4. The data processing method of claim 1, wherein before the step of determining whether the field length of the data set is less than or equal to a data transmission threshold, the data processing method further comprises: and performing data cleaning on the target data in the data set according to the data attribute corresponding to the target data and the field length threshold, wherein the data attribute indicates the service type corresponding to the target data.
5. The data processing method of claim 4, further comprising: and judging whether the target data subjected to data cleaning is variable-length field data, if so, adjusting the field length of the data set after the data cleaning.
6. The data processing method according to claim 1, wherein in case that the field length of the data set is greater than a data transmission threshold, the data processing method further comprises: and triggering a current limiting rule, and stopping executing the caching operation on the data set.
7. A data processing apparatus, comprising:
the target data acquisition module is used for acquiring target data in real time and carrying out serialization processing on the target data;
the field length determining module is used for placing the target data subjected to serialization processing in a data set and determining the field length corresponding to the data set;
the judging module is used for judging whether the field length corresponding to the data set exceeds a data transmission threshold value or not;
and the processing module is used for executing cache operation on the data set and transmitting the data set executing the cache operation according to a transmission protocol under the condition that the length of the field corresponding to the data set is less than or equal to a data transmission threshold value.
8. The data processing apparatus of claim 7, wherein the field length determination module is further configured to: dividing the target data into fixed-length field data and variable-length field data according to the data type, determining field length accuracy according to the field length of the fixed-length field data and the field length of the variable-length field data, and taking the field length corresponding to the variable-length field data as the field length corresponding to the data set under the condition that the field length accuracy is greater than or equal to an accuracy threshold.
9. An electronic device, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-6.
10. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010564944.7A CN113760630B (en) | 2020-06-19 | 2020-06-19 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010564944.7A CN113760630B (en) | 2020-06-19 | 2020-06-19 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113760630A true CN113760630A (en) | 2021-12-07 |
| CN113760630B CN113760630B (en) | 2024-09-20 |
Family
ID=78785492
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010564944.7A Active CN113760630B (en) | 2020-06-19 | 2020-06-19 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113760630B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070088705A1 (en) * | 2005-10-13 | 2007-04-19 | Yen-Fu Chen | System and method for capture and processing of overflow characters from user input |
| WO2012146649A1 (en) * | 2011-04-26 | 2012-11-01 | Robert Bosch Gmbh | Method and device for serial data transmission which is adapted to memory sizes |
| CN108289125A (en) * | 2018-01-26 | 2018-07-17 | 华南理工大学 | TCP sessions recombination based on Stream Processing and statistical data extracting method |
| CN110928905A (en) * | 2019-11-07 | 2020-03-27 | 泰康保险集团股份有限公司 | Data processing method and device |
| CN111182488A (en) * | 2019-12-05 | 2020-05-19 | 诺得物流股份有限公司 | Traceability data energy-saving transmission method based on time channel |
-
2020
- 2020-06-19 CN CN202010564944.7A patent/CN113760630B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070088705A1 (en) * | 2005-10-13 | 2007-04-19 | Yen-Fu Chen | System and method for capture and processing of overflow characters from user input |
| WO2012146649A1 (en) * | 2011-04-26 | 2012-11-01 | Robert Bosch Gmbh | Method and device for serial data transmission which is adapted to memory sizes |
| CN108289125A (en) * | 2018-01-26 | 2018-07-17 | 华南理工大学 | TCP sessions recombination based on Stream Processing and statistical data extracting method |
| CN110928905A (en) * | 2019-11-07 | 2020-03-27 | 泰康保险集团股份有限公司 | Data processing method and device |
| CN111182488A (en) * | 2019-12-05 | 2020-05-19 | 诺得物流股份有限公司 | Traceability data energy-saving transmission method based on time channel |
Non-Patent Citations (1)
| Title |
|---|
| 卢晓旭等: "LHAASO地面簇射粒子阵列在线实时分布式数据处理", 核技术, no. 04, 15 April 2020 (2020-04-15) * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113760630B (en) | 2024-09-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109684358B (en) | Data query method and device | |
| CN111124819B (en) | Method and device for full link monitoring | |
| CN113517985B (en) | File data processing method and device, electronic equipment and computer readable medium | |
| CN112966200A (en) | Service degradation method and rendering server | |
| CN113760982A (en) | Data processing method and device | |
| CN112084042A (en) | Message processing method and device | |
| CN113742389A (en) | Service processing method and device | |
| CN111800223B (en) | Method, device and system for generating sending message and processing receiving message | |
| CN110933122B (en) | Method, apparatus and computer storage medium for managing server | |
| CN113760630B (en) | Data processing method and device | |
| CN114374657B (en) | Data processing method and device | |
| CN110247847B (en) | Method and device for back source routing between nodes | |
| CN112988417B (en) | Message processing method, device, electronic equipment and computer readable medium | |
| CN113761433B (en) | Service processing method and device | |
| CN110019671B (en) | Method and system for processing real-time message | |
| CN113722193A (en) | Method and device for detecting page abnormity | |
| CN113779018A (en) | Data processing method and device | |
| CN113765813A (en) | Method and device for dynamically adjusting message receiving rate | |
| CN117979210B (en) | Audio transmission method, device, electronic equipment and storage medium | |
| CN113760980B (en) | Data caching method, data providing end and data using end | |
| CN113238808A (en) | Message pushing method and device | |
| CN112788075A (en) | Business service monitoring method and device | |
| CN110852537A (en) | Service quality detection method and device | |
| CN113132480B (en) | Data transmission method, device and system | |
| US10571998B2 (en) | Systems and methods for operating computing devices at peak power efficiency in real time |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |