CN111211864B - Data transmission error processing method and system - Google Patents
Data transmission error processing method and system Download PDFInfo
- Publication number
- CN111211864B CN111211864B CN201911360065.6A CN201911360065A CN111211864B CN 111211864 B CN111211864 B CN 111211864B CN 201911360065 A CN201911360065 A CN 201911360065A CN 111211864 B CN111211864 B CN 111211864B
- Authority
- CN
- China
- Prior art keywords
- data
- characteristic information
- source
- sink
- extracting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L1/00—Arrangements for detecting or preventing errors in the information received
- H04L1/004—Arrangements for detecting or preventing errors in the information received by using forward error control
- H04L1/0041—Arrangements at the transmitter end
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/64—Protecting data integrity, e.g. using checksums, certificates or signatures
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L1/00—Arrangements for detecting or preventing errors in the information received
- H04L1/004—Arrangements for detecting or preventing errors in the information received by using forward error control
- H04L1/0045—Arrangements at the receiver end
Abstract
The invention discloses a data transmission error processing method and a system, a rolling copy is created for a data set of a data sink, a time stamp is added when source data is transmitted, all changes of the data set in a target data sink after the time stamp are recorded as data changes, first characteristic information and second characteristic information are compared by extracting the first characteristic information of the source data and the second characteristic information of transmitted data, when the second characteristic information is not consistent with the first characteristic information, the data set is restored to a time stamp node according to the data changes, when data transmission is in error, the rolling copy can be used for restoring the data set of the data sink, the irreversible problem caused by the pollution of the data sink or the input of error data due to the data transmission error is effectively prevented, and the integrity and the correctness of the data sink are ensured.
Description
Technical Field
The present invention relates to the field of data transmission technologies, and in particular, to a method and a system for processing data transmission errors.
Background
Data transmission is a communication process of transmitting data from a data source to a data sink, and can be divided into parallel transmission and serial transmission according to the data transmission sequence, synchronous transmission and asynchronous transmission according to a synchronous mode, and errors can be caused in data transmission due to external interference, signal distortion and the like in the data transmission process;
The existing processing method is to delete the error data when the data has errors, and re-request the data source to send the data, but for many data sinks, such as the data sink in the neural network, once the data arrives, the data set in the data sink may be polluted, even if the error data is deleted, the data sink may be irreversibly affected, and the final data deviation is caused.
Disclosure of Invention
In view of the above, the present invention provides a data error processing method and system, which can process data transmission errors more effectively.
Based on the above purpose, the invention provides a data transmission error processing method, which comprises the following steps:
creating a copy of the film for the data set of the data sink;
when source data are sent, time stamps are respectively added to the data set and the volume copy in the data sink;
recording all changes of the data set in the target data sink after the timestamp in the copy of the film file, and recording the changes as data changes;
extracting first characteristic information of source data;
when the source data is delivered to the target data sink, extracting second characteristic information of the delivered data;
comparing the first characteristic information and the second characteristic information;
And if the second characteristic information is not consistent with the first characteristic information, restoring the data set to the timestamp node according to the data change.
Preferably, extracting the first characteristic information of the source data includes:
acquiring the file size of source data;
and calculating the distance of a preset data cluster in the source data.
Preferably, the extracting the second feature information of the delivery data includes:
acquiring the file size of delivery data;
and calculating the distance of a preset data cluster in the delivered data.
Preferably, the preset distance of the data clusters conforms to the following formula
Wherein d is the distance between the ith data cluster and the jth data cluster, and X i For the ith data cluster, X j Is the jth data cluster.
Preferably, before restoring the data set to the timestamp node according to the data change, the method further includes:
recording the error times;
if the error times exceed the preset times, executing data reduction;
otherwise, the source data is obtained from the data source again;
and extracting the first characteristic information and comparing the first characteristic information with the second characteristic information again.
A data transmission error handling system comprising:
the copy module is used for creating a volume copy for the data set of the data sink;
The time recording module is used for respectively adding timestamps to the data set in the data sink and the film copy when the source data is sent;
the data change recording module records all changes of the database in the target data sink after the timestamp in the copy of the film file and records the changes as data changes;
the characteristic extraction module is used for extracting first characteristic information of the source data and extracting second characteristic information of the delivery data when the source data is delivered to the target data sink;
the characteristic comparison module is used for comparing the first characteristic information with the second characteristic information;
and the data restoration module restores the data set to the timestamp node according to the data change if the second characteristic information is inconsistent with the first characteristic information.
Preferably, the first characteristic information includes a file size of the source data and a distance of a preset data cluster in the source data;
the second characteristic information comprises the size of the delivered data and the distance of a preset data cluster in the delivered data.
Preferably, the method further comprises the following steps:
and the error recording module is used for recording the error times when the second characteristic information is inconsistent with the first characteristic information, and restoring the data set to the timestamp node according to the data change when the error times exceed the set times.
From the above, it can be seen that, in the data transmission error processing method and system provided by the present invention, a volume copy is created for a data set of a data sink, a timestamp is added when source data is transmitted, all changes occurring in the data set in a target data sink after the timestamp are recorded as data changes, first characteristic information and second characteristic information are compared by extracting the first characteristic information of the source data and the second characteristic information of the transmission data, and when the second characteristic information is not consistent with the first characteristic information, the data set is restored to a timestamp node according to the data changes.
Drawings
FIG. 1 is a flowchart illustrating an error handling method according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to specific embodiments and the accompanying drawings.
It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are used for distinguishing two entities with the same name but different names or different parameters, and it should be noted that "first" and "second" are merely for convenience of description and should not be construed as limitations of the embodiments of the present invention, and they are not described in any more detail in the following embodiments.
A data transmission error processing method, as shown in fig. 1, includes the following steps:
s101, creating a volume copy for a data set of a data sink;
the data sink is the receiver in the data transmission, and the complete information of the data set of the data sink is recorded in the film copy during creation.
S102, when source data is sent, time stamps are respectively added to the data set in the data sink and the film copy;
the added time stamp is used for recording the time of source data transmission, and the time stamps in the data sink and the film copy should be synchronous.
S103, recording all changes of the data set in the target data sink after the timestamp in the copy of the film, and recording the changes as data changes;
s104, extracting first characteristic information of the source data;
s105, when the source data is delivered to the target data sink, extracting second characteristic information of the delivered data;
s106, comparing the first characteristic information with the second characteristic information;
s107, if the second characteristic information is not consistent with the first characteristic information, restoring the data set to the timestamp node according to the data change.
The invention discloses a data transmission error processing method, which comprises the steps of creating a volume copy for a data set of a data sink, adding a timestamp when source data is transmitted, marking all changes of the data set in a target data sink after the timestamp as data changes, comparing first characteristic information with second characteristic information by extracting the first characteristic information of the source data and the second characteristic information of transmitted data, reducing the data set to a timestamp node according to the data changes when the second characteristic information is inconsistent with the first characteristic information, reducing the data set of the data sink by using the volume copy when data transmission is in error, effectively preventing the data sink from being polluted or the irreversible problem caused by the input of error data due to data transmission errors, and ensuring the integrity and the correctness of the data sink.
As an embodiment, extracting the first characteristic information of the source data includes:
acquiring the file size of source data;
and calculating the distance of a preset data cluster in the source data.
By presetting the data cluster, any continuous data combination which appears more than once in the source data can be in the data cluster, and when the occurrence frequency is more, the average value or the histogram distribution of each distance can be calculated.
As one embodiment, extracting the second feature information of the delivery data includes:
acquiring the file size of delivery data;
and calculating the distance of a preset data cluster in the delivered data.
By extracting the first characteristic information and the second characteristic information, whether the source data and the delivered data have difference or not, namely whether data transmission errors occur or not can be judged rapidly and accurately, and because errors occurring in the data transmission process are usually format errors, data loss, abnormal data difference values and the like, the delivered data files are different in size or the distance between data clusters is changed.
As an embodiment, the preset distance of the data clusters conforms to the following formula:
wherein d is the distance between the ith data cluster and the jth data cluster, Xi is the ith data cluster, and Xj is the jth data cluster.
As an embodiment, before the data change restores the data set to the timestamp node, the method further includes:
recording the error times;
if the error times exceed the preset times, executing data reduction;
re-acquiring source data from a data source;
and extracting the first characteristic information and comparing the first characteristic information with the second characteristic information again.
By reacquiring the source data, recording the error times, and performing data recovery when the error times exceed the preset times, retransmission can be directly allowed when the user judges that the output transmission error does not pollute the data sink, and the user can also set the preset times to 0.
The invention also provides a data transmission error processing system, comprising:
the copy module is used for creating a volume copy for the data set of the data sink;
the time recording module is used for respectively adding timestamps to the data set in the data sink and the film copy when the source data is sent;
the data change recording module records all changes of the database in the target data sink after the timestamp in the copy of the film file and records the changes as data changes;
the characteristic extraction module is used for extracting first characteristic information of the source data and extracting second characteristic information of the delivery data when the source data is delivered to the target data sink;
The characteristic comparison module is used for comparing the first characteristic information with the second characteristic information;
and the data restoration module restores the data set to the timestamp node according to the data change if the second characteristic information is inconsistent with the first characteristic information.
The method comprises the steps of creating a volume copy for a data set of a data sink, adding a timestamp when source data are sent, recording all changes of the data set in a target data sink after the timestamp as data changes, comparing first characteristic information with second characteristic information by extracting the first characteristic information of the source data and the second characteristic information of the sent data, and when the second characteristic information is not consistent with the first characteristic information, restoring the data set to a timestamp node according to the data changes.
As an embodiment, the first characteristic information includes a file size of the source data and a preset distance of the data cluster in the source data;
The second characteristic information includes the size of the delivered data and the distance of a preset data cluster in the delivered data.
As an implementation manner, the system further includes an error recording module, configured to record the number of errors when the second feature information does not match the first feature information, and restore the data set to the timestamp node according to the data change when the number of errors exceeds a set number.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, is limited to these examples; within the idea of the invention, also features in the above embodiments or in different embodiments may be combined, steps may be implemented in any order, and there are many other variations of the different aspects of the invention as described above, which are not provided in detail for the sake of brevity.
In addition, well known power/ground connections to Integrated Circuit (IC) chips and other components may or may not be shown within the provided figures for simplicity of illustration and discussion, and so as not to obscure the invention. Furthermore, devices may be shown in block diagram form in order to avoid obscuring the invention, and also in view of the fact that specifics with respect to implementation of such block diagram devices are highly dependent upon the platform within which the present invention is to be implemented (i.e., specifics should be well within purview of one skilled in the art). Where specific details (e.g., circuits) are set forth in order to describe example embodiments of the invention, it should be apparent to one skilled in the art that the invention can be practiced without, or with variation of, these specific details. Accordingly, the description is to be regarded as illustrative instead of restrictive.
While the present invention has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art in light of the foregoing description. For example, other memory architectures (e.g., dynamic ram (dram)) may use the discussed embodiments.
The embodiments of the invention are intended to embrace all such alternatives, modifications and variances that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, substitutions, improvements and the like that may be made without departing from the spirit and principles of the invention are intended to be included within the scope of the invention.
Claims (5)
1. A method for handling data transmission errors, the method comprising:
creating a copy of the film for the data set of the data sink;
when source data are sent, time stamps are respectively added to the data set and the volume copy in the data sink;
recording all changes of the data set in the target data sink after the timestamp in the copy of the film file, and recording the changes as data changes;
extracting first characteristic information of source data;
when the source data is delivered to the target data sink, extracting second characteristic information of the delivered data;
Comparing the first characteristic information and the second characteristic information;
if the second characteristic information is not consistent with the first characteristic information, restoring the data set to the timestamp node according to the data change;
the extracting first characteristic information of the source data comprises:
acquiring the file size of source data;
calculating the distance of a preset data cluster in the source data;
the extracting of the second feature information of the delivery data comprises:
acquiring the file size of delivery data;
and calculating the distance of a preset data cluster in the delivered data.
2. The data transmission error processing method of claim 1, wherein the predetermined distance between the data clusters satisfies the following formula
Wherein d is the distance between the ith data cluster and the jth data cluster, and X i For the ith data cluster, X j Is the jth data cluster.
3. The data transmission error handling method of claim 1, wherein before restoring the data set to the timestamp node according to the data change, further comprising:
recording the error times;
if the error times exceed the preset times, executing data reduction;
Otherwise, the source data is obtained from the data source again;
and extracting the first characteristic information and comparing the first characteristic information with the second characteristic information.
4. A data transmission error handling system, comprising:
the copy module is used for creating a volume copy for the data set of the data sink;
the time recording module is used for respectively adding timestamps to the data set in the data sink and the film copy when the source data is sent;
the data change recording module records all changes of the data set in the target data sink after the timestamp in the copy of the film file and records the changes as data changes;
the characteristic extraction module is used for extracting first characteristic information of the source data and extracting second characteristic information of the delivery data when the source data is delivered to the target data sink;
the characteristic comparison module is used for comparing the first characteristic information with the second characteristic information;
the data restoration module restores the data set to the timestamp node according to the data change if the second characteristic information is inconsistent with the first characteristic information;
the first characteristic information comprises the file size of the source data and the distance of a preset data cluster in the source data;
The second characteristic information comprises the size of the delivered data and the distance of a preset data cluster in the delivered data.
5. The data transmission error handling system of claim 4, further comprising:
and the error recording module is used for recording the error times when the second characteristic information is inconsistent with the first characteristic information, and restoring the data set to the timestamp node according to the data change when the error times exceed the set times.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911360065.6A CN111211864B (en) | 2019-12-25 | 2019-12-25 | Data transmission error processing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911360065.6A CN111211864B (en) | 2019-12-25 | 2019-12-25 | Data transmission error processing method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111211864A CN111211864A (en) | 2020-05-29 |
| CN111211864B true CN111211864B (en) | 2022-07-29 |
Family
ID=70789342
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911360065.6A Active CN111211864B (en) | 2019-12-25 | 2019-12-25 | Data transmission error processing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111211864B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106339274A (en) * | 2015-07-14 | 2017-01-18 | 阿里巴巴集团控股有限公司 | Method and system for obtaining data snapshot |
| CN107835983A (en) * | 2015-04-16 | 2018-03-23 | 诺宝公司 | Backup-and-restore is carried out in distributed data base using consistent database snapshot |
| CN108804464A (en) * | 2017-05-03 | 2018-11-13 | 中兴通讯股份有限公司 | Data consistency detection, device, distributed data base and storage medium |
| CN109597965A (en) * | 2018-11-19 | 2019-04-09 | 深圳力维智联技术有限公司 | Data processing method, system, terminal and medium based on deep neural network |
| CN110598065A (en) * | 2019-08-28 | 2019-12-20 | 腾讯云计算（北京）有限责任公司 | Data mining method and device and computer readable storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10157183B2 (en) * | 2015-07-28 | 2018-12-18 | International Business Machines Corporation | Real time and scalable space-time data recovery |
-
2019
- 2019-12-25 CN CN201911360065.6A patent/CN111211864B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107835983A (en) * | 2015-04-16 | 2018-03-23 | 诺宝公司 | Backup-and-restore is carried out in distributed data base using consistent database snapshot |
| CN106339274A (en) * | 2015-07-14 | 2017-01-18 | 阿里巴巴集团控股有限公司 | Method and system for obtaining data snapshot |
| CN108804464A (en) * | 2017-05-03 | 2018-11-13 | 中兴通讯股份有限公司 | Data consistency detection, device, distributed data base and storage medium |
| CN109597965A (en) * | 2018-11-19 | 2019-04-09 | 深圳力维智联技术有限公司 | Data processing method, system, terminal and medium based on deep neural network |
| CN110598065A (en) * | 2019-08-28 | 2019-12-20 | 腾讯云计算（北京）有限责任公司 | Data mining method and device and computer readable storage medium |
Non-Patent Citations (2)
| Title |
|---|
| 基于可变时间窗口的增量数据抽取模型;刘杰等;《计算机科学》;20181115;全文 * |
| 改进Dijkstra算法在电动汽车充电申请中的研究与实现;王飞，王国义，袁涛;《宁夏师范学院学报》;20180715;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111211864A (en) | 2020-05-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109902071B (en) | Service log storage method, system, device and equipment | |
| CN110475124B (en) | Video jamming detection method and device | |
| CN110113393B (en) | Message pushing method and device, electronic equipment and medium | |
| CN110061843B (en) | Block height creating method, device and equipment in chain type account book | |
| CN103064757A (en) | Method and system for backing up data | |
| CN111355549B (en) | Data protection method and device | |
| CN105099729A (en) | User ID (Identification) recognition method and device | |
| WO2022082891A1 (en) | Big data acquisition method and system, and computer device and storage medium thereof | |
| CN111211864B (en) | Data transmission error processing method and system | |
| CN112380229A (en) | Service data synchronization method and device, nonvolatile storage medium and processor | |
| US10664349B2 (en) | Method and device for file storage | |
| CN108418746B (en) | Mail synchronization method and device and computer readable storage medium | |
| CN109861843B (en) | Method, device and equipment for completely collecting and confirming log files | |
| CN108256311B (en) | Authorization authentication method and device and electronic equipment | |
| CN112787873B (en) | IOAM time delay measurement performance sequencing method and system | |
| CN110858790A (en) | Data packet transmission method and device, storage medium and electronic device | |
| CN116841801A (en) | Data backup method and system based on cloud server | |
| CN111757371A (en) | Statistical method of transmission delay, server and storage medium | |
| CN106293996A (en) | Method of data synchronization, main equipment, alternate device and system | |
| CN114297318A (en) | Data processing method and device | |
| CN112597210A (en) | Data summarizing method, device, equipment and storage medium | |
| CN113037420A (en) | Reading time stamp obtaining method and device, electronic equipment and storage medium | |
| CN107920279A (en) | The method and apparatus for assessing video quality | |
| CN112527787B (en) | Safe and reliable multiparty data deduplication system, method and device | |
| EP3223478A1 (en) | Packet processing method and device, and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |