CN111800512A - Data transmission method and server - Google Patents
Data transmission method and server Download PDFInfo
- Publication number
- CN111800512A CN111800512A CN202010646575.6A CN202010646575A CN111800512A CN 111800512 A CN111800512 A CN 111800512A CN 202010646575 A CN202010646575 A CN 202010646575A CN 111800512 A CN111800512 A CN 111800512A
- Authority
- CN
- China
- Prior art keywords
- data
- task data
- original
- server
- backup
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/104—Peer-to-peer [P2P] networks
- H04L67/1074—Peer-to-peer [P2P] networks for supporting data block transmission mechanisms
- H04L67/1078—Resource delivery mechanisms
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application discloses a data transmission method and a server, which are used for ensuring the idempotency of data service when the server provides the data service for a client. The method in the embodiment of the application comprises the following steps: and the server side assembles n original task data to obtain target task data, and backups the target task data to obtain backup task data. When the client requests data again, the server can extract the data required by the client from the backup task data and provide the data to the client, so that the same data can be provided to the client every time the client requests the data, and the idempotency of data service is ensured.
Description
Technical Field
The embodiment of the application relates to the field of data transmission, in particular to a data transmission method and a server side.
Background
When executing a data processing task, a client needs to request data from a server, and the process is that the client firstly sends a data downloading request to the server to request the server to issue the data; after receiving the data downloading request, the server side performs data preparation, namely, acquires original data to be issued to the client side, assembles the acquired original data to obtain assembled target data, and returns the target data to the client side.
In the data request mode, in order to protect the privacy of the client and the security of the data, after the target data is returned to the client, the server deletes the target data and the original data corresponding to the target data, so that the privacy of the client is prevented from being leaked due to the leakage of the target data and the original data.
However, after the original data and the target data are both deleted, when the client needs to request the data again, the original data and the target data are both deleted, so the server cannot return the same target data to the client any more, which results in that the data download service provided by the server has no idempotent.
Disclosure of Invention
The embodiment of the application provides a data transmission method and a server, which are used for ensuring the idempotency of data service when the server provides the data service for a client.
A first aspect of the embodiments of the present application provides a data transmission method, which is applied to a server, and the method includes:
receiving a data request sent by a client, wherein the data request is used for requesting the server to send target task data;
responding to the data request, and acquiring n original task data, wherein each original task data is a sub-item in the target task data, and n is a positive integer;
assembling the n original task data into the target task data;
backing up the target task data to obtain backup task data, wherein the backup task data is used for providing the backup task data in subsequent data requests of the client;
and responding to the data request, and sending the target task data to the client.
Preferably, the method further comprises:
receiving processing progress information sent by the client when the client finishes data processing on the target task data, wherein the processing progress information is used for representing the current processing progress of the target task data;
and deleting the backup task data according to the processing progress information.
Preferably, the backing up the target task data includes:
backing up the target task data in a cache to obtain the backup task data, and setting the valid period of the backup task data;
the deleting the backup task data according to the processing progress information includes:
and based on the cached effective storage mechanism, deleting the backup task data when the valid period of the backup task data is reached according to the processing progress information.
Preferably, each original task data corresponds to one identification information, and the identification information between each original task data is different;
the assembling the n original task data into the target task data includes:
respectively taking the identification information of each original task data as a key and the original task data corresponding to the key as a value to form n key value pair data;
and assembling the n key value pair data into the target task data.
Preferably, the n pieces of identification information of the n pieces of original task data are n serial numbers with a permutation order;
the assembling the n key-value pair data into the target task data comprises:
sorting the n key values according to the arrangement sequence of the n serial numbers;
and assembling the n sorted key value pairs into the target task data.
Preferably, after the target task data is sent to the client, the method further includes:
receiving a subsequent data request of the client, wherein the subsequent data request is used for requesting the server to send m original task data in the backup task data, m is a positive integer and is not more than n;
responding to the subsequent data request, and extracting the m original task data corresponding to the subsequent data request from the backup task data;
and assembling the m original task data into data to be sent, and sending the data to be sent to the client.
Preferably, the subsequent data request includes a serial number corresponding to each original task data in the m original task data, and the m serial numbers have an arrangement order;
the extracting the m original task data corresponding to the subsequent data request from the backup task data includes:
extracting original task data corresponding to m serial numbers respectively from the backup task data to obtain m original task data;
the assembling the m original task data into data to be sent includes:
respectively taking each serial number in the m serial numbers as a key, and taking original task data corresponding to each key as a value to form m key value pair data;
sorting the m key values according to the arrangement sequence of the m serial numbers;
and assembling the sorted m key value pairs into the data to be sent.
A second aspect of the embodiments of the present application provides a data transmission method, which is applied to a client, and the method includes:
sending a data request to a server, wherein the data request is used for requesting the server to send target task data;
receiving the target task data sent by the server based on the data request, and performing data processing on the target task data, wherein the target task data is obtained by assembling n original task data by the server, and n is a positive integer;
and when the data processing of the target task data is finished, the target task data is not stored and is discarded.
Preferably, the method further comprises:
and when the data processing of the target task data is finished, asynchronously sending processing progress information to the server, wherein the processing progress information is used for representing the current processing progress of the target task data, so that the server deletes the backup task data according to the processing progress information.
Preferably, the data processing of the target task data includes:
performing data processing on the target task data, and when data loss occurs in the target task data, sending a subsequent data request to the server, wherein the subsequent data request is used for requesting the server to send m original task data in backup task data, and the backup task data is backup of the target task data, wherein m is a positive integer and is less than or equal to n;
and receiving the m original task data sent by the server based on the subsequent data request, and performing data processing by combining the m original task data.
A third aspect of the embodiments of the present application provides a server, where the server includes:
the system comprises a receiving unit, a sending unit and a processing unit, wherein the receiving unit is used for receiving a data request sent by a client, and the data request is used for requesting the server to send target task data;
an obtaining unit, configured to respond to the data request, and obtain n pieces of original task data, where each piece of original task data is a sub-item in the target task data, and n is a positive integer;
the assembling unit is used for assembling the n original task data into the target task data;
the backup unit is used for backing up the target task data to obtain backup task data, and the backup task data is used for providing the backup task data in subsequent data requests of the client;
and the sending unit is used for responding to the data request and sending the target task data to the client.
Preferably, the receiving unit is further configured to receive processing progress information sent by the client when the client completes data processing on the target task data, where the processing progress information is used to indicate a current processing progress of the target task data;
the server side further comprises:
and the deleting unit is used for deleting the backup task data according to the processing progress information.
Preferably, the backup unit is specifically configured to backup the target task data in a cache, obtain the backup task data, and set a validity period of the backup task data;
the deleting unit is specifically configured to delete the backup task data when the validity period of the backup task data is reached based on the cached valid storage mechanism according to the processing progress information.
Preferably, each original task data corresponds to one identification information, and the identification information between each original task data is different;
the assembling unit is specifically configured to take the identification information of each piece of original task data as a key, take the original task data corresponding to the key as a value, form n key value pair data, and assemble the n key value pair data into the target task data.
Preferably, the n pieces of identification information of the n pieces of original task data are n serial numbers with a permutation order;
the assembling unit is specifically configured to sort the n key value pairs of data according to the arrangement order of the n serial numbers, and assemble the sorted n key value pairs of data into the target task data.
Preferably, the receiving unit is further configured to receive a subsequent data request of the client, where the subsequent data request is used to request the server to send m original task data in the backup task data, where m is a positive integer and m is not greater than n;
the obtaining unit is further configured to extract, in response to the subsequent data request, the m original task data corresponding to the subsequent data request from the backup task data;
the assembling unit is further configured to assemble the m original task data into data to be sent, and send the data to be sent to the client.
Preferably, the subsequent data request includes a serial number corresponding to each original task data in the m original task data, and the m serial numbers have an arrangement order;
the obtaining unit is specifically configured to extract original task data corresponding to m serial numbers in the backup task data, to obtain the m original task data;
the assembling unit is specifically configured to take each of the m serial numbers as a key, take original task data corresponding to each key as a value, form m key value pair data, sort the m key value pair data according to an arrangement order of the m serial numbers, and assemble the sorted m key value pair data into the data to be sent.
A fourth aspect of an embodiment of the present application provides a client, where the client includes:
the system comprises a sending unit, a receiving unit and a processing unit, wherein the sending unit is used for sending a data request to a server, and the data request is used for requesting the server to send target task data;
a receiving unit, configured to receive the target task data sent by the server based on the data request;
the data processing unit is used for carrying out data processing on the target task data, the target task data is obtained by assembling n original task data by the server, wherein n is a positive integer;
and the execution unit is used for not storing the target task data and discarding the target task data when the data processing of the target task data is finished.
Preferably, the sending unit is further configured to asynchronously send processing progress information to the server when the data processing of the target task data is completed, where the processing progress information is used to indicate a current processing progress of the target task data, so that the server deletes the backup task data according to the processing progress information.
Preferably, the data processing unit is specifically configured to perform data processing on the target task data, and when data loss occurs in the target task data, send a subsequent data request to the server, where the subsequent data request is used to request the server to send m original task data in backup task data, and the backup task data is a backup of the target task data, where m is a positive integer and m is not greater than n;
the receiving unit is further configured to receive the m original task data sent by the server based on the subsequent data request;
the data processing unit is specifically configured to perform data processing in combination with the m original task data.
A fifth aspect of the embodiments of the present application provides a server, including:
a processor, a memory, an input and output device;
the processor is connected with the memory and the input and output equipment;
the input and output equipment is used for receiving a data request sent by a client, and the data request is used for requesting the server to send target task data;
the processor is configured to respond to the data request, acquire n original task data, where each original task data is a sub-item in the target task data, where n is a positive integer, assemble the n original task data into the target task data, and back up the target task data to obtain backup task data, where the backup task data is used to provide the backup task data in a subsequent data request of the client;
the input and output device is further used for responding to the data request and sending the target task data to the client.
A sixth aspect of embodiments of the present application provides a computer storage medium having stored therein instructions that, when executed on a computer, cause the computer to perform the method of the first aspect.
According to the technical scheme, the embodiment of the application has the following advantages:
and the server side assembles n original task data to obtain target task data, and backups the target task data to obtain backup task data. When the client requests data again, the server can extract the data required by the client from the backup task data and provide the data to the client, so that the same data can be provided to the client every time the client requests the data, and the idempotency of data service is ensured.
Drawings
Fig. 1 is a schematic flow chart of a data transmission method in an embodiment of the present application;
fig. 2 is another schematic flow chart illustrating a data transmission method according to an embodiment of the present application;
fig. 3 is another schematic flow chart illustrating a data transmission method according to an embodiment of the present application;
FIG. 4 is a schematic structural diagram of a server in an embodiment of the present application;
FIG. 5 is a schematic structural diagram of a client in an embodiment of the present application;
fig. 6 is another schematic structural diagram of a server in the embodiment of the present application.
Detailed Description
The embodiment of the application provides a data transmission method and a server, which are used for ensuring the idempotency of data service when the server provides the data service for a client.
Referring to fig. 1, an embodiment of a data transmission method in the embodiment of the present application includes:
101. receiving a data request sent by a client;
in this embodiment, the client may need to perform one or more data processing tasks. When executing a data processing task, a client needs to acquire target task data corresponding to the data processing task from a server, so as to perform data processing on the acquired target task data. At this time, the client sends a data request to the server, and the server receives the data request.
102. Responding to the data request, and acquiring n original task data;
when a client performs a data processing task, the data processing task may be directed to one or more items of data. For example, when calculating the profit, it is necessary to acquire a plurality of items of data such as the turnover and the operating cost. Therefore, the target task data corresponding to the data processing task also includes one or more items of data.
Therefore, after receiving the data request, the server responds to the data request to obtain n original task data, each original task data is a sub-item in the target task data, that is, the target task data includes n original task data, where n is a positive integer.
In this embodiment, the server may obtain the n original task data from the database, may also obtain the n original task data from the local storage, or obtain the n original task data from other nodes, and a specific obtaining manner is not limited.
103. Assembling n original task data into target task data;
in order to facilitate the transmission of the target task data, after acquiring n original task data, the server assembles the n original task data into the target task data required by the client. At this time, the target task data is equivalent to an array, n original task data form n elements in the array, and the n elements can be arranged in order, so that the target task data can be conveniently processed by the client.
104. Backing up the target task data to obtain backup task data;
after the target task data is obtained through assembly, in order that the client can obtain the same original task data in subsequent data requests, the server needs to backup the target task data to obtain backup task data. Therefore, in subsequent data requests, the server can extract the same original task data from the backup task data and provide the same original task data to the client, so that the idempotency of the data service provided by the server is ensured.
105. Responding to the data request, and sending target task data to the client;
after the target task data is assembled, the server responds to a data request of the client and sends the target task data to the client, so that the client can execute a task of data processing according to the target task data.
In this embodiment, the server side assembles n original task data to obtain target task data, and backs up the target task data to obtain backup task data. When the client requests data again, the server can extract the data required by the client from the backup task data and provide the data to the client, so that the same data can be provided to the client every time the client requests the data, and the idempotency of data service is ensured.
The above describes the process of the server sending the target task data to the client, and the following further describes the operation performed by the client after receiving the target task data. Referring to fig. 2, another embodiment of a data transmission method according to the embodiment of the present application includes:
201. sending a data request to a server;
when the data processing task is executed, the client sends a data request to the server so as to conveniently acquire target task data corresponding to the data processing task and perform data processing on the acquired target task data.
202. Receiving target task data sent by a server based on a data request, and performing data processing on the target task data;
after the server receives the data request of the client, the server assembles n original task data to obtain target task data, and sends the target task data to the client, so that the client can perform data processing on the received target task data. Wherein n is a positive integer.
203. Not storing the target task data and discarding the target task data;
in order to prevent the target task data from occupying the storage resources of the client and save the management cost of the client on the target task data, when the client finishes the data processing of the target task data, the client does not store the target task data and discards the target task data, so as to save the storage space of the client.
Embodiments of the present application will be described in further detail below on the basis of the aforementioned embodiments shown in fig. 1 to 2. Referring to fig. 3, another embodiment of the data transmission method in the embodiment of the present application includes:
301. a server receives a data request sent by a client;
302. the server side responds to the data request and obtains n original task data;
the operations performed in steps 301 to 302 are similar to the operations performed in steps 101 to 102 in the embodiment shown in fig. 1, and are not described again here.
303. The server side assembles n original task data into target task data;
in this embodiment, the server may perform any form of assembly on the original task data. In one embodiment, each piece of original task data corresponds to one piece of identification information, and the identification information is different between each piece of original task data. When the original task data are assembled, the identification information of each original task data is used as a key, the original task data corresponding to each key is used as a value, and then n key value pair data are formed. The n key-value pairs of data may then be assembled into target task data.
For example, seq (×) is used as identification information of the original task data, seq1 is identification information of the original task data 1, and seq2 is identification information of the original task data 2. Using seq1 as the key for original task data 1 and seq2 as the key for original task data 2, key-value pair data seq1: value1 and seq2: value2 can be formed, respectively, where value1 refers to original task data 1 and value2 refers to original task data 2. By analogy, n key value pairs of data may be formed.
It should be noted that the identification information of the original task data may be in any form, for example, it may be letters, numbers or other literal symbols, as long as it can be used to identify a unique original task data, and the specific form is not limited herein.
Specifically, the identification information may be a serial number, and the n pieces of identification information of the n pieces of original task data may be n serial numbers in the arrangement order. Therefore, after the n key value pair data are obtained, the n key value pair data can be sorted according to the arrangement order of the n serial numbers, and the sorted n key value pair data can be assembled into target task data.
For example, following the above example, the n pieces of identification information may be n pieces of identification information sequentially arranged in order of seq1, seq2, … …, and seq, and then n pieces of key-value pair data of seq1: value1, seq2: value2, … …, and seq: value may be formed. Then, the n key value pair data are sorted according to the arrangement order of the identification information corresponding to the n key value pair data, and the sorted n key value pair data are assembled into target task data, so that the target task data are [ seq1: value1, seq2: value2, seq3: value3, …, and seq: value ].
By forming key value pair data and orderly assembling the original task data, the client can quickly identify and disassemble multiple subdata in the target task data when receiving the target task data, and the client can conveniently process the target task data.
In addition, the assembly manner of the original task data by the server may be an unordered assembly manner, that is, a plurality of original task data do not need to be assembled in any order, and the assembly manner is not limited herein as long as the plurality of original task data can be assembled to obtain the target task data.
304. The server side backs up the target task data to obtain backup task data;
in this embodiment, in order to facilitate obtaining the backup task data, the server may backup the target task data in the cache, that is, establish backup of the target task data in the cache.
The cache may be a Redi s cache, a Memcached cache, or an Ehcache cache, or a Tair cache, and a specific cache type is not limited, preferably a Redis cache.
In addition, in order to avoid occupying too much storage space in the cache when the backup task data is continuously increased, the server may set a validity period of the backup task data, and delete the corresponding backup task data when the validity period is reached.
305. The server side responds to the data request and sends target task data to the client side;
the operation performed in this step is similar to the operation performed in step 105 in the embodiment shown in fig. 1, and is not repeated here.
306. The client performs data processing on the target task data;
and after receiving the target task data, the client performs data processing on the target task data. The specific way of data processing may be various operations on the target task data, such as data calculation, data analysis, or data comparison, and the specific manner is not limited.
307. The client sends a subsequent data request to the server;
in the data processing process of the target task data, when abnormal conditions such as poor network conditions, shutdown and restart of control equipment of the client occur, part or all of the data in the target task data may be lost. In this case, since the server has already established the backup of the target task data, the client may request the server for data again, that is, send a subsequent data request to the server, where the subsequent data request is used to request the server to send m original task data in the backup task data, where m is a positive integer and m is less than or equal to n, that is, the data requested by the subsequent data request may be part or all of the original task data in the backup task data.
308. The server side responds to the subsequent data request, and m original task data corresponding to the subsequent data request are extracted from the backup task data;
after receiving a subsequent data request of the client, the server extracts m original task data corresponding to the subsequent data request from the backup task data.
Specifically, the subsequent data request of the client may include a sequence number corresponding to each of the m pieces of original task data, where the m pieces of sequence numbers have an arrangement order. Therefore, the server can search the original task data corresponding to the m serial numbers in the backup task data according to the m serial numbers, and finally extract the m original task data.
For example, following the example in step 303, after obtaining the target task data [ seq1: value1, seq2: value2, seq3: value3, …, seq: value ], the target task data is backed up, and the obtained backup task data are also [ seq1: value1, seq2: value2, seq3: value3, …, seq: value ]. The subsequent data request of the client comprises the serial numbers of m original task data, the sequence of the m serial numbers is seq1, seq3, seq4, seq8, …, and seq, and the server searches the corresponding m original task data in the backup task data according to the m serial numbers, so that the original task data corresponding to seq1 to seq are obtained.
309. The server side assembles the m original task data into data to be sent;
after extracting m original task data corresponding to the subsequent data request, the server assembles the m original task data, and the assembled m original task data may be referred to as data to be sent.
Specifically, m key value pair data may be formed by taking each of m serial numbers of the m pieces of original task data as a key and taking the original task data corresponding to each key as a value. And then, sequencing the m key value data according to the sequence of the m serial numbers, and assembling the sequenced m key value data into data to be sent.
For example, following the example given in step 308, m key value pair data are formed by taking each of the m sequence numbers as a key and the original task data corresponding to each key as a value when the original task data corresponding to seq1 to seq are obtained, and are seq1: value1, seq3: value3, seq4: value4, seq8: value8, …, and seq m: value, respectively. Then, the m key value data are sorted according to the arrangement order of the m serial numbers, and the sorted m key value data are assembled into data to be sent, so that the data to be sent are obtained as [ seq1: value1, seq3: value3, seq4: value4, seq8: value8, …, and seq: value ].
310. The client receives m original task data sent by the server based on the subsequent data request, and performs data processing by combining the m original task data;
after the server side acquires data to be sent including m original task data based on a subsequent data request of the client side, the data to be sent is sent to the client side, so that the client side can receive the m original task data, and continue to execute a data processing task by combining the m original task data, so as to complete data processing of target task data.
311. Not storing the target task data and discarding the target task data;
the operation performed in this step is similar to the operation performed in step 203 in the embodiment shown in fig. 2, and is not described again here.
312. The client sends processing progress information to the server;
when the data processing of the target task data is completed, the client sends processing progress information to the server, wherein the processing progress information represents the current processing progress of the target task data, namely represents that the target task data is processed currently, and is equivalent to that the client notifies the server that the target task data is processed completely.
The client may send the processing progress information synchronously or asynchronously. The asynchronous transmission has the advantages that if the network is in a bad condition when the data processing is finished, the client can normally transmit the processing progress information after the network recovers after a period of time, and the information does not need to be transmitted immediately when the data processing is finished, so that the network resources can be effectively utilized, and the failure of information transmission is avoided.
It should be noted that, when the data processing of the target task data is completed, the client may first perform step 311, and then perform step 312; step 312 may be performed first, and then step 311 may be performed. And is not particularly limited herein.
313. The server deletes the backup task data according to the processing progress information;
after receiving the processing progress information sent by the client, the server can determine that the target task data has completed data processing. At this time, it is not necessary to keep the backup of the target task data (i.e. backup task data) for the client. Therefore, the server deletes the backup task data according to the received processing progress information.
The effective storage mechanism of the cache means that the data stored in the cache is automatically cleaned and deleted by the cache when the expiration date is reached. Therefore, in this embodiment, if the backup task data is stored in the cache, since the valid period of the backup task data is set in the cache, the backup task data reaching the valid period can be automatically deleted when it is known that the target task data completes the data processing according to the processing progress information based on the valid storage mechanism of the cache, so as to save the storage space of the cache.
In addition, the server can also directly and manually delete the backup task data, namely, when the target task data is determined to complete the data processing, the server manually deletes the backup task data.
In this embodiment, when abnormal conditions such as data loss occur, the client may send a subsequent data request to the server to request backup of the same original task data in the task data, thereby further ensuring idempotent of data service of the server.
The above describes the data transmission method in the embodiment of the present application, and the following describes the server in the embodiment of the present application, referring to fig. 4, an embodiment of the server in the embodiment of the present application includes:
a receiving unit 401, configured to receive a data request sent by a client, where the data request is used to request a server to send target task data;
an obtaining unit 402, configured to respond to a data request, and obtain n pieces of original task data, where each piece of original task data is a sub-item in target task data, and n is a positive integer;
an assembling unit 403, configured to assemble the n original task data into target task data;
a backup unit 404, configured to backup target task data to obtain backup task data, where the backup task data is used to provide backup task data in a subsequent data request of a client;
a sending unit 405, configured to send target task data to the client in response to the data request.
In a preferred implementation manner of this embodiment, the receiving unit 401 is further configured to receive processing progress information sent by the client when the data processing on the target task data is completed, where the processing progress information is used to indicate a current processing progress of the target task data;
the server side also comprises:
a deleting unit 406, configured to delete the backup task data according to the processing progress information.
In another preferred embodiment of this embodiment, the backup unit 404 is specifically configured to backup target task data in a cache, obtain backup task data, and set an expiration date of the backup task data;
the deleting unit 406 is specifically configured to delete the backup task data when the validity period of the backup task data is reached according to the processing progress information based on the valid storage mechanism of the cache.
In another preferred implementation manner of this embodiment, each piece of original task data corresponds to one piece of identification information, and the identification information between each piece of original task data is different;
the assembling unit 403 is specifically configured to use the identification information of each original task data as a key, use the original task data corresponding to the key as a value, form n key value pair data, and assemble the n key value pair data into the target task data.
In another preferred implementation manner of this embodiment, n pieces of identification information of n pieces of original task data are n serial numbers having a permutation order;
the assembling unit 403 is specifically configured to sort the n key value pairs of data according to the arrangement order of the n serial numbers, and assemble the sorted n key value pairs of data into target task data.
In another preferred implementation manner of this embodiment, the receiving unit 401 is further configured to receive a subsequent data request of the client, where the subsequent data request is used to request the server to send m original task data in the backup task data, where m is a positive integer, and m is less than or equal to n;
the obtaining unit 402 is further configured to, in response to the subsequent data request, extract m original task data corresponding to the subsequent data request from the backup task data;
the assembling unit 403 is further configured to assemble the m original task data into data to be sent, and send the data to be sent to the client.
In another preferred implementation manner of this embodiment, the subsequent data request includes a serial number corresponding to each of m original task data, and the m serial numbers have an arrangement order;
the obtaining unit 402 is specifically configured to extract original task data corresponding to the m serial numbers in the backup task data, to obtain m original task data;
the assembling unit 403 is specifically configured to use each serial number of the m serial numbers as a key, use the original task data corresponding to each key as a value, form m key value pair data, sort the m key value pair data according to the arrangement order of the m serial numbers, and assemble the sorted m key value pair data into data to be sent.
In this embodiment, operations performed by each unit in the server are similar to those described in the embodiments shown in fig. 1 and fig. 3, and are not described again here.
In this embodiment, the assembling unit 403 obtains target task data after assembling n original task data, and the backup unit 404 backs up the target task data to obtain backup task data. When the client requests data again, the obtaining unit 402 may extract data required by the client from the backup task data and provide the data to the client, thereby ensuring that the same data can be provided to the client each time the client requests data, and ensuring idempotent of data service.
Referring to fig. 5, an embodiment of the client in the embodiment of the present application includes:
a sending unit 501, configured to send a data request to a server, where the data request is used to request the server to send target task data;
a receiving unit 502, configured to receive target task data sent by a server based on a data request;
a data processing unit 503, configured to perform data processing on target task data, where the target task data is obtained by assembling n original task data by a server, where n is a positive integer;
the execution unit 504 is configured to, when the data processing of the target task data is completed, not store the target task data and discard the target task data.
In a preferred embodiment of this embodiment, the sending unit 501 is further configured to asynchronously send processing progress information to the server when the data processing of the target task data is completed, where the processing progress information is used to indicate a current processing progress of the target task data, so that the server deletes the backup task data according to the processing progress information.
In another preferred embodiment of this embodiment, the data processing unit 503 is specifically configured to perform data processing on target task data, and when data loss occurs in the target task data, send a subsequent data request to the server, where the subsequent data request is used to request the server to send m original task data in backup task data, and the backup task data is a backup of the target task data, where m is a positive integer and m is less than or equal to n;
the receiving unit 502 is further configured to receive m original task data sent by the server based on the subsequent data request;
the data processing unit 503 is specifically configured to perform data processing in combination with the m pieces of original task data.
In this embodiment, operations performed by each unit in the client are similar to those described in the embodiments shown in fig. 2 and fig. 3, and are not described again here.
Referring to fig. 6, a service end in an embodiment of the present application is described below, where an embodiment of the service end in the embodiment of the present application includes:
the server 600 may include one or more Central Processing Units (CPUs) 601 and a memory 605, where the memory 605 stores one or more applications or data.
The memory 605 may be volatile storage or persistent storage, among other things. The program stored in the memory 605 may include one or more modules, each of which may include a sequence of instructions operating on a server. Further, the central processor 601 may be configured to communicate with the memory 605 to execute a series of instruction operations in the memory 605 on the server 600.
The server 600 may also include one or more power supplies 602, one or more wired or wireless network interfaces 603, one or more input-output interfaces 604, and/or one or more operating systems, such as Windows Server, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.
The central processing unit 601 can execute the operations executed by the server in the embodiments shown in fig. 1 and fig. 3, and details thereof are not repeated herein.
An embodiment of the present application further provides a computer storage medium, where one embodiment includes: the computer storage medium stores instructions that, when executed on a computer, cause the computer to perform the operations performed by the server in the embodiments shown in fig. 1 and 3.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, a division of a unit is merely a logical division, and an actual implementation may have another division, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
Units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method of the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and the like.
Claims (12)
1. A data transmission method is applied to a server side, and the method comprises the following steps:
receiving a data request sent by a client, wherein the data request is used for requesting the server to send target task data;
responding to the data request, and acquiring n original task data, wherein each original task data is a sub-item in the target task data, and n is a positive integer;
assembling the n original task data into the target task data;
backing up the target task data to obtain backup task data, wherein the backup task data is used for providing the backup task data in subsequent data requests of the client;
and responding to the data request, and sending the target task data to the client.
2. The data transmission method of claim 1, further comprising:
receiving processing progress information sent by the client when the client finishes data processing on the target task data, wherein the processing progress information is used for representing the current processing progress of the target task data;
and deleting the backup task data according to the processing progress information.
3. The data transmission method according to claim 2, wherein the backing up the target task data comprises:
backing up the target task data in a cache to obtain the backup task data, and setting the valid period of the backup task data;
the deleting the backup task data according to the processing progress information includes:
and based on the cached effective storage mechanism, deleting the backup task data when the valid period of the backup task data is reached according to the processing progress information.
4. The data transmission method according to claim 1, wherein each of the original task data corresponds to an identification information, and the identification information is different between each of the original task data;
the assembling the n original task data into the target task data includes:
respectively taking the identification information of each original task data as a key and the original task data corresponding to the key as a value to form n key value pair data;
and assembling the n key value pair data into the target task data.
5. The data transmission method according to claim 4, wherein the n pieces of identification information of the n pieces of original task data are n serial numbers having a permutation order;
the assembling the n key-value pair data into the target task data comprises:
sorting the n key values according to the arrangement sequence of the n serial numbers;
and assembling the n sorted key value pairs into the target task data.
6. The data transmission method according to any one of claims 1 to 5, wherein after the sending of the target task data to the client, the method further comprises:
receiving a subsequent data request of the client, wherein the subsequent data request is used for requesting the server to send m original task data in the backup task data, m is a positive integer and is not more than n;
responding to the subsequent data request, and extracting the m original task data corresponding to the subsequent data request from the backup task data;
and assembling the m original task data into data to be sent, and sending the data to be sent to the client.
7. The data transmission method according to claim 6, wherein the subsequent data request includes a sequence number corresponding to each of the m original task data, and the m sequence numbers have a permutation order;
the extracting the m original task data corresponding to the subsequent data request from the backup task data includes:
extracting original task data corresponding to m serial numbers respectively from the backup task data to obtain m original task data;
the assembling the m original task data into data to be sent includes:
respectively taking each serial number in the m serial numbers as a key, and taking original task data corresponding to each key as a value to form m key value pair data;
sorting the m key values according to the arrangement sequence of the m serial numbers;
and assembling the sorted m key value pairs into the data to be sent.
8. A data transmission method is applied to a client, and the method comprises the following steps:
sending a data request to a server, wherein the data request is used for requesting the server to send target task data;
receiving the target task data sent by the server based on the data request, and performing data processing on the target task data, wherein the target task data is obtained by assembling n original task data by the server, and n is a positive integer;
and when the data processing of the target task data is finished, the target task data is not stored and is discarded.
9. The data transmission method of claim 8, wherein the method further comprises:
and when the data processing of the target task data is finished, asynchronously sending processing progress information to the server, wherein the processing progress information is used for representing the current processing progress of the target task data, so that the server deletes the backup task data according to the processing progress information.
10. The data transmission method according to claim 8, wherein the performing data processing on the target task data includes:
performing data processing on the target task data, and when data loss occurs in the target task data, sending a subsequent data request to the server, wherein the subsequent data request is used for requesting the server to send m original task data in backup task data, and the backup task data is backup of the target task data, wherein m is a positive integer and is less than or equal to n;
and receiving the m original task data sent by the server based on the subsequent data request, and performing data processing by combining the m original task data.
11. A server, characterized in that the server comprises:
the system comprises a receiving unit, a sending unit and a processing unit, wherein the receiving unit is used for receiving a data request sent by a client, and the data request is used for requesting the server to send target task data;
an obtaining unit, configured to respond to the data request, and obtain n pieces of original task data, where each piece of original task data is a sub-item in the target task data, and n is a positive integer;
the assembling unit is used for assembling the n original task data into the target task data;
the backup unit is used for backing up the target task data to obtain backup task data, and the backup task data is used for providing the backup task data in subsequent data requests of the client;
and the sending unit is used for responding to the data request and sending the target task data to the client.
12. A server, comprising:
a processor, a memory, an input and output device;
the processor is connected with the memory and the input and output equipment;
the input and output equipment is used for receiving a data request sent by a client, and the data request is used for requesting the server to send target task data;
the processor is configured to respond to the data request, acquire n original task data, where each original task data is a sub-item in the target task data, where n is a positive integer, assemble the n original task data into the target task data, and back up the target task data to obtain backup task data, where the backup task data is used to provide the backup task data in a subsequent data request of the client;
the input and output device is further used for responding to the data request and sending the target task data to the client.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010646575.6A CN111800512B (en) | 2020-07-07 | 2020-07-07 | Data transmission method and server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010646575.6A CN111800512B (en) | 2020-07-07 | 2020-07-07 | Data transmission method and server |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111800512A true CN111800512A (en) | 2020-10-20 |
| CN111800512B CN111800512B (en) | 2023-08-04 |
Family
ID=72810329
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010646575.6A Active CN111800512B (en) | 2020-07-07 | 2020-07-07 | Data transmission method and server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111800512B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103838762A (en) * | 2012-11-26 | 2014-06-04 | 中国银联股份有限公司 | Uninterrupted data processing device and method based on data table rotation |
| CN106921712A (en) * | 2015-12-28 | 2017-07-04 | 阿里巴巴集团控股有限公司 | A kind of method for processing business and device |
-
2020
- 2020-07-07 CN CN202010646575.6A patent/CN111800512B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103838762A (en) * | 2012-11-26 | 2014-06-04 | 中国银联股份有限公司 | Uninterrupted data processing device and method based on data table rotation |
| CN106921712A (en) * | 2015-12-28 | 2017-07-04 | 阿里巴巴集团控股有限公司 | A kind of method for processing business and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111800512B (en) | 2023-08-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107819891B (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN106251202A (en) | Maliciously order recognition methods and device | |
| CN112506702B (en) | Disaster recovery method, device, equipment and storage medium for data center | |
| CN110968478A (en) | Log collection method, server and computer storage medium | |
| US11531676B2 (en) | Method and system for anomaly detection based on statistical closed-form isolation forest analysis | |
| CN110196759B (en) | Distributed transaction processing method and device, storage medium and electronic device | |
| CN111111214B (en) | Game archiving processing method, device and system | |
| CN111177601B (en) | Page rendering processing method, device, equipment and readable storage medium | |
| CN111818117A (en) | Data updating method and device, storage medium and electronic equipment | |
| CN108880885B (en) | Message processing method and device | |
| CN113051102B (en) | File backup method, device, system, storage medium and computer equipment | |
| US9237074B1 (en) | Distributed identifier generation system | |
| CN111526207A (en) | Data transmission method and related equipment | |
| CN114844771A (en) | Monitoring method, device, storage medium and program product for micro-service system | |
| CN111800512A (en) | Data transmission method and server | |
| CN113298620A (en) | Frequent traveler information management method and related device | |
| CN104991814A (en) | Task execution result acquiring method and system | |
| CN116821896A (en) | Malicious sample file acquisition system, malicious sample file reporting method and device | |
| CN112579833B (en) | Service association relation acquisition method and device based on user operation data | |
| CN110134547B (en) | Middleware-based repeated data deleting method and related device | |
| CN111800409B (en) | Interface attack detection method and device | |
| WO2014184799A2 (en) | Workflow execution system and method for cloud environment | |
| CN113157494B (en) | Method and device for backing up data in block chain system | |
| CN113419936B (en) | Dynamic point burying method, device, equipment and storage medium | |
| CN112835531B (en) | Data pushing method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |