CN115085869A - Data transmission method, device, equipment and storage medium - Google Patents
Data transmission method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN115085869A CN115085869A CN202210761802.9A CN202210761802A CN115085869A CN 115085869 A CN115085869 A CN 115085869A CN 202210761802 A CN202210761802 A CN 202210761802A CN 115085869 A CN115085869 A CN 115085869A
- Authority
- CN
- China
- Prior art keywords
- data
- content
- polling
- cache
- target file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L1/00—Arrangements for detecting or preventing errors in the information received
- H04L1/08—Arrangements for detecting or preventing errors in the information received by repeating transmission, e.g. Verdan system
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L1/00—Arrangements for detecting or preventing errors in the information received
- H04L1/20—Arrangements for detecting or preventing errors in the information received using signal quality detector
Abstract
The application discloses a data transmission method, a device, equipment and a storage medium, wherein the method comprises the following steps: in the process of transmitting data to be transmitted, after the data content of a data fragment is transmitted to a target file, acquiring the distribution condition of illegal characters of the content stored in the target file; the data to be transmitted is segmented into a plurality of data segments; the storage content corresponds to the data content; and if the distribution condition meets a preset data damage condition, determining that the stored content is damaged, and retransmitting the data content of the data fragment to the target file. The method and the device realize that the waste of network resources and computing resources is reduced during data transmission.
Description
Technical Field
The present application relates to the field of data transmission, and in particular, to a data transmission method, apparatus, device, and storage medium.
Background
When data transmission is performed, there may be a case where the transmitted data is damaged. Currently, for a situation that data corruption may exist, it is necessary to manually find the situation that data corruption exists after data is completely transmitted, or determine that data corruption exists when a program reads access data and throws an exception.
After determining that there is data corruption, the source data needs to be retransmitted to ensure that the transmitted data can be used normally. However, the data transmission process consumes network resources and computing resources, and especially some data are very large files, which greatly wastes network resources and computing resources.
Disclosure of Invention
In view of this, embodiments of the present application provide a data transmission method, apparatus, device and storage medium, which aim to reduce waste of network resources and computing resources during data transmission.
In order to achieve the above object, the present application provides a data transmission method, where the method includes:
in the process of transmitting data to be transmitted, after the data content of a data fragment is transmitted to a target file, acquiring the distribution condition of illegal characters of the content stored in the target file; the data to be transmitted is segmented into a plurality of data segments; the storage content corresponds to the data content;
and if the distribution condition meets a preset data damage condition, determining that the stored content is damaged, and retransmitting the data content of the data fragment to the target file.
For example, before obtaining the distribution of the illegal characters of the content stored in the target file, the method includes:
polling a cache pipeline of the data transmission assembly to obtain a polling result; the data transmission assembly is used for transmitting data to be transmitted;
if the polling result is that cache data exists in the cache pipeline, transmitting the data content of one data fragment in the cache data to a target file; the cache data comprises at least one data slice.
Illustratively, the polling the cache pipe of the data transmission component to obtain the polling result includes:
acquiring the data category of the data to be transmitted;
determining a number of polls to poll the data transmission component based on the data category;
and polling the cache pipeline of the data transmission assembly based on the polling times to obtain a polling result.
For example, if the polling result indicates that the cache data exists in the cache pipeline, transmitting the data content of one data fragment in the cache data to the target file includes:
if the polling result is that cache data exists in the cache pipeline, determining whether a target file exists in a data receiver;
if yes, determining whether an available data block exists in the target file;
and if so, writing the data content of one data fragment in the cache data into the available data block.
For example, if yes, after writing the data content of one data slice in the cache data into the available data block, the method includes:
starting a file input stream; the file input stream is used for inputting the storage content in the available data blocks;
if the file input stream fails to be started, deleting the storage content corresponding to the data fragments in the available data blocks, and returning the polling times of the cache pipeline to the polling times when the polling is started;
and retransmitting the data content corresponding to the deleted storage content to the available data block.
Illustratively, after obtaining the distribution of the illegal characters of the content stored in the target file, the method further includes:
if the distribution condition does not meet the preset data damage condition, closing the file write stream;
if the closing of the file write stream fails, deleting the storage content corresponding to the data fragment in the available data block, and returning the polling times of the cache pipeline to the polling times when the polling is started;
and retransmitting the data content corresponding to the deleted storage content to the available data block.
For example, after determining that the storage content is damaged and retransmitting the data content of the data segment to the target file if the distribution condition meets a preset data damage condition, the method includes:
and if the polling times of polling the cache pipeline are equal to the preset polling times or the cache data are not polled from the cache pipeline, closing the daemon agent for guarding the data transmission process in the current round.
In addition, to achieve the above object, the present application also provides a data transmission apparatus, including:
the acquisition module is used for acquiring the distribution condition of illegal characters of the content stored in a target file after the data content of a data fragment is transmitted to the target file in the process of transmitting the data to be transmitted; the data to be transmitted is segmented into a plurality of data segments; the storage content corresponds to the data content;
and the determining module is used for determining that the storage content is damaged if the distribution condition meets a preset data damage condition, and retransmitting the data content of the data fragment to the target file.
In addition, to achieve the above object, the present application also provides a data transmission device, which includes a memory, a processor, and a data transmission program stored on the memory and operable on the processor, and when executed by the processor, implements the steps of the data transmission method as described above.
In addition, to achieve the above object, the present application also provides a computer readable storage medium having a data transmission program stored thereon, which when executed by a processor implements the steps of the data transmission method as described above.
Compared with the prior art that whether data are damaged or not can be determined after all data are transmitted, and the data are retransmitted after the data are damaged, so that a great deal of waste of network resources and computing resources is caused. By means of the method and the device for transmitting the data, the data content corresponding to the damaged part of the storage content can be retransmitted as long as the storage content is found to be damaged in the process of transmitting the data to be transmitted by means of segmenting the data, and the retransmission of all the data after the data is completely transmitted is not needed, so that the waste of network resources and computing resources is reduced during data transmission.
Drawings
Fig. 1 is a schematic flow chart of a first embodiment of the data transmission method of the present application;
FIG. 2 is a schematic flow chart of data transmission in a first embodiment of the data transmission method of the present application;
fig. 3 is a schematic diagram of a daemon process of a daemon agent in the first embodiment of the data transmission method of the present application;
fig. 4 is a schematic structural diagram of a hardware operating environment according to an embodiment of the present application.
The implementation, functional features and advantages of the objectives of the present application will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
Referring to fig. 1, fig. 1 is a schematic flow chart of a first embodiment of the data transmission method.
While embodiments of the data transmission method are provided in the present application, it should be noted that, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown or described here. The data transmission method can be applied to a server.
It should be noted that the method is implemented based on the flume, the flume is a data acquisition and transmission assembly and mainly comprises three units, namely a source unit, a channel unit and a sink unit, wherein the source unit is used for acquiring data from a source file, the channel unit is used for caching the data when the data size of the data acquired by the source unit is equal to a preset data size, and the sink unit is used for transmitting the data to a target file. Wherein the data includes offline data and real-time data.
It should be noted that the source may obtain data in the source file in batches, and the data size of each batch of data is a preset data size. The preset data size may be set as required, and this embodiment is not particularly limited.
Referring to fig. 2, fig. 2 illustrates a process of transferring data in a source file to a destination file via a flash, wherein a data pipeline is a channel, and a sink is not shown in the figure.
It should be noted that the steps of the present application are performed at a stage after the data is submitted to the channel.
It should be noted that, for convenience of description, the execution of the respective steps of the subject description data transmission method is omitted below. The data transmission method comprises the following steps:
step S10, in the process of transmitting data to be transmitted, after the data content of a data fragment is transmitted to a target file, the distribution condition of illegal characters of the content stored in the target file is obtained; the data to be transmitted is divided into a plurality of data fragments; the stored content corresponds to the data content.
Illustratively, the data to be transmitted is data in a cache pipe (channel), and the data to be transmitted needs to be written from the source file into the destination file.
For example, the target file may be a file established in a local disk for storing data in the source file, or may be a file established in a database for storing data in the source file.
For example, the process of transmitting the data content to the target file may be directly storing the original text, or encoding the data content in a specified encoding format and then storing the encoded data content. For storage after encoding in a specified encoding format, generally, the encoding format is a compression format, for example, if data content needs to be stored in a target file in a compression format, metadata needs to be written in a corresponding data block of the target file through a file write stream, the compression format of the metadata is identified, and finally, the data content is compressed based on the compression format to obtain compressed data, and the compressed data is written in the data block; of course, the encoding format may be other formats, for example, the data content is video, and the conversion between the two video formats, mp4 and wmv, is also the encoding process. It is understood that the data content becomes the storage content after being stored in the target file.
Note that the data size of each data block in the target file is fixed, for example, 2048B, 3072B, and the like.
Illustratively, the obtaining the distribution of the illegal characters of the content stored in the target file includes:
step a, if the storage content is subjected to encoding processing, acquiring an encoding format of the storage content;
and b, decoding the storage content based on the coding format to obtain decoded storage content.
If the stored content is stored in the original text of the data content, decoding is not needed; and if the storage content is obtained after the data content is coded, obtaining the coding format, and decoding the storage content to obtain the decoded storage content.
And c, acquiring the distribution condition of illegal characters in the decoding storage content.
The memory content is a byte sequence, and correspondingly, the decoding memory content is also a byte sequence. It should be noted that the distribution of illegal characters in the byte sequence has a specific expression, for example, for the byte sequence encoded by "unicode", when the character "\ uffffd" appears in the byte sequence, it is determined that the illegal character exists in the byte sequence.
For example, the distribution of the illegal characters may be obtained from the acquired storage content after the storage content is obtained from the target file, or may be directly obtained from the target file.
For example, before obtaining the distribution of the illegal characters of the content stored in the target file, the method includes:
d, polling the cache pipeline of the data transmission assembly to obtain a polling result; the data transmission assembly is used for transmitting data to be transmitted.
A plurality of data fragments, namely a batch of data acquired by the source, are cached in the cache pipeline.
It should be noted that, after a batch of data acquired by the source is submitted to the cache pipeline, a daemon agent for guarding the data transmission process of the batch of data is started, and the daemon agent is used for reading data contents in the data fragments from the data pipeline and determining whether the storage contents are damaged. Referring to fig. 3, fig. 3 is a schematic diagram of the daemon agent in the embodiment.
Before the daemon agent reads the data content in the data fragment from the data pipeline, whether cache data still exist in the cache pipeline needs to be determined through a data sniffer, wherein the data sniffer is started after the daemon agent is started.
It should be noted that polling the cache pipeline is implemented by a data sniffer, and when the data sniffer sniffs that the cache data in the cache pipeline is empty, or the sniffing frequency reaches the polling frequency (the sniffing frequency is increased by one every time the data sniffer sniffs the cache pipeline), the data sniffer is turned off to stop polling.
Illustratively, the polling the cache pipe of the data transmission component to obtain the polling result includes:
and d1, acquiring the data type of the data to be transmitted.
The data category comprises offline and real-time, namely the data to be transmitted comprises offline data and real-time data.
Step d2, determining the polling times for polling the data transmission component based on the data category.
Different data classes, which have different requirements for data transmission. The number of data fragments of each batch of data is inversely proportional to timeliness and directly proportional to transmission efficiency, and the number of data fragments is the same as the polling frequency, namely the process of polling the data fragments in the cache pipeline by the data sniffer is substantially equivalent to traversing the data fragments in the cache pipeline. For offline, it does not require timeliness of transmission, but requires higher transmission efficiency, so when the data category is offline, the polling times are set to be larger, for example, 100 times, 110 times, etc.; for real time, it requires timeliness of transmission but does not require transmission efficiency, and therefore, when the data category is offline, the number of polling times is set to be small, for example, 20 times, 30 times, and the like.
And d3, polling the cache pipeline of the data transmission assembly based on the polling times to obtain a polling result.
The polling result includes the presence of cached data in the cache pipe and the absence of cached data in the cache pipe. After the data sniffer sniffs the cache data each time, the daemon agent reads the cache data and successfully writes the data content of the cache data into the target file, the cache data transmitted this time can be deleted, so that the repeated transmission of the data is avoided.
It will be appreciated that if no cache data is present in the cache pipe, then polling is stopped.
Step e, if the polling result is that cache data exists in the cache pipeline, transmitting the data content of one data fragment in the cache data to a target file; the cache data comprises at least one data slice.
For example, if the polling result indicates that the cache data exists in the cache pipeline, transmitting the data content of one data fragment in the cache data to the target file includes:
step e1, if the polling result is that the cache data exists in the cache pipeline, determining whether the target file exists in the data receiver.
The data receiver can be a file system in a server, a remote database, etc.
Step e2, if yes, determining whether there is available data block in the target file;
and e3, if yes, writing the data content of one data fragment in the cache data into the available data block.
Referring to fig. 2, when there is an available data block in the target file, the data content may be stored in the available data block, and if there is no available data block in the target file, a new data block is created in the target file and the data content is stored in the new data block. In addition, if the target file cannot be found in the data receiver, the target file is created first, and then a new data block is created in the target file.
For example, if yes, after writing the data content of one data slice in the cache data into the available data block, the method includes:
step e31, opening a file input stream; the file input stream is used for inputting the storage content in the available data blocks.
The file input stream is used for inputting the storage content in the available data blocks and loading the storage content into the memory.
Step e32, if the file input stream fails to be opened, deleting the storage content corresponding to the data fragment in the available data block, and returning the polling frequency of the cache pipeline to the polling frequency at the beginning of the polling.
The failure of starting the file input stream is represented as outputting prompt information or reporting an exception, the prompt information is used for prompting that the user data cannot be loaded, and the prompt information is recorded in a log file. In addition, after the file input stream fails to be opened, it indicates that reading data from the data block fails, that is, the storage content in the data block cannot be read normally, and at this time, the data in the data block needs to be deleted. Moreover, since the data content of the data segment of this time is not successfully transmitted to the target file, a backoff time is required, so that the polling time is backed up to the polling time before the last sniffing, that is, the process of transmitting the data content of the data segment of this time is completely reset.
And e33, retransmitting the data content corresponding to the deleted storage content to the available data block.
It should be noted that the specific implementation of retransmitting the data content to the data block is basically the same as the specific implementation of transmitting the cache data, and is not described herein again.
Step S20, if the distribution condition meets a preset data damage condition, determining that the stored content is damaged, and retransmitting the data content of the data fragment to the target file.
And if the distribution condition does not meet the preset data damage condition, determining that the stored content is not damaged.
The preset data corruption condition may be set as required, for example, when 2 "\ uFFFD" or 3 "\ uFFFD" appears in the byte sequence, the data corruption is determined, wherein "\ uFFFD" may be arranged continuously or discontinuously.
It should be noted that the embodiment of retransmitting the data content of the data fragment to the target file is substantially the same as the specific embodiment of retransmitting the data content to the data block, and is not described herein again. It will be appreciated that re-transmitting the data fragments enables the corrupted memory content to be automatically repaired immediately after the memory content is found to be corrupted.
In addition, whether the stored content is damaged or not can be found in the process of fragment transmission (transmission is carried out after the stored content is fragmented into a plurality of data fragments), so that the condition that the data to be transmitted is lost because the stored content is not found to be damaged in time and the opportunity of retransmission is missed is avoided. Wherein, the case of missing the opportunity of retransmission includes: longer, source files cleaned, etc.
Illustratively, after obtaining the distribution of the illegal characters of the content stored in the target file, the method further includes:
and f, if the distribution condition does not meet the preset data damage condition, closing the file write stream.
If the distribution condition does not meet the preset data damage condition, the process of writing the data into the target file is determined to be basically finished, and the file write-in stream can be closed at the moment.
And g, if the closing of the file write stream fails, deleting the storage content corresponding to the data fragment in the available data block, and returning the polling times of the cache pipeline to the polling times when the polling is started.
If the file write stream fails to be closed, it indicates that an error exists in the data transmission process, and the storage content corresponding to the data fragment of the current transmission, which has been written into the available data block, needs to be deleted.
It should be noted that the specific implementation of the backoff number of this time is substantially the same as the specific implementation of the backoff number when the file input stream fails to be opened, and details thereof are not repeated herein.
And h, retransmitting the data content corresponding to the deleted storage content to the available data block.
The embodiment of this step is substantially the same as that of step e33, and is not described herein again.
For example, after determining that the storage content is damaged and retransmitting the data content of the data segment to the target file if the distribution condition meets a preset data damage condition, the method includes:
and i, if the polling times of polling the cache pipeline are equal to the preset polling times or the cache data are not polled from the cache pipeline, closing the daemon agent for guarding the data transmission process of the current round.
And after the cache data does not exist in the cache pipeline, closing the daemon agent, and starting the transmission process of the next batch of data.
Compared with the prior art that whether data are damaged or not can be determined after all data are transmitted, and the data are retransmitted after the data are damaged, so that a great deal of waste of network resources and computing resources is caused. By means of the method and the device for transmitting the data, the data content corresponding to the damaged part of the storage content can be retransmitted as long as the storage content is found to be damaged in the process of transmitting the data to be transmitted by means of segmenting the data, and the retransmission of all the data after the data is completely transmitted is not needed, so that the waste of network resources and computing resources is reduced during data transmission.
In addition, this application still provides a data transmission device, the device includes:
the acquisition module is used for acquiring the distribution condition of illegal characters of the content stored in a target file after the data content of a data fragment is transmitted to the target file in the process of transmitting the data to be transmitted; the data to be transmitted is segmented into a plurality of data segments; the storage content corresponds to the data content;
and the determining module is used for determining that the storage content is damaged if the distribution condition meets a preset data damage condition, and retransmitting the data content of the data fragment to the target file.
Illustratively, the data transmission apparatus further includes:
the polling module is used for polling the cache pipeline of the data transmission assembly to obtain a polling result; the data transmission component is used for transmitting data to be transmitted;
the transmission module is used for transmitting the data content of one data fragment in the cache data to a target file if the polling result indicates that the cache data exists in the cache pipeline; the cache data comprises at least one data slice.
Illustratively, the polling module is specifically configured to:
acquiring the data category of the data to be transmitted;
determining a polling number of polling of a data transmission component based on the data category;
and polling the cache pipeline of the data transmission assembly based on the polling times to obtain a polling result.
Illustratively, the transmission module is specifically configured to:
if the polling result is that cache data exists in the cache pipeline, determining whether a target file exists in a data receiver;
if yes, determining whether an available data block exists in the target file;
and if so, writing the data content of one data fragment in the cache data into the available data block.
Illustratively, the transmission module is further configured to:
starting a file input stream; the file input stream is used for inputting the storage content in the available data blocks;
if the file input stream fails to be started, deleting the storage content corresponding to the data fragments in the available data blocks, and returning the polling times of the cache pipeline to the polling times when the polling is started;
and retransmitting the data content corresponding to the deleted storage content to the available data block.
Illustratively, the data transmission apparatus further includes:
a closing module, configured to close a file write stream if the distribution condition does not meet a preset data damage condition;
a deleting module, configured to delete the storage content corresponding to the data fragment in the available data block if the file write stream is failed to close, and rollback the polling times of the cache pipeline to the polling times when the polling is started;
and the retransmission module is used for retransmitting the data content corresponding to the deleted storage content to the available data block.
Illustratively, the data transmission apparatus further includes:
and the closing module is used for closing the daemon agent for guarding the data transmission process of the current round if the polling times of polling the cache pipeline are equal to the preset polling times or the cache data are not polled from the cache pipeline.
Illustratively, the second obtaining module is specifically configured to:
if the storage content is subjected to encoding processing, acquiring an encoding format of the storage content;
decoding the storage content based on the coding format to obtain decoded storage content;
and acquiring the distribution condition in the decoding storage content.
The specific implementation of the data transmission device of the present application is substantially the same as the embodiments of the data transmission method, and is not described herein again.
In addition, the application also provides data transmission equipment. As shown in fig. 4, fig. 4 is a schematic structural diagram of a hardware operating environment according to an embodiment of the present application.
It should be noted that fig. 4 is a schematic structural diagram of a hardware operating environment of the data transmission device.
As shown in fig. 4, the data transmission apparatus may include: a processor 4001, for example, a CPU, a memory 4005, a user interface 4003, a network interface 4004, and a communication bus 4002. The communication bus 4002 is used to realize connection communication among these components. The user interface 4003 may include a Display (Display), an input unit such as a Keyboard (Keyboard), and the optional user interface 4003 may also include a standard wired interface, a wireless interface. Network interface 4004 may optionally include a standard wired interface, a wireless interface (e.g., a WI-FI interface). The memory 4005 may be a high-speed RAM memory or a non-volatile memory (e.g., a disk memory). The memory 4005 may alternatively be a storage device separate from the processor 4001.
Optionally, the data transmission device may further include RF (Radio Frequency) circuits, sensors, audio circuits, WiFi modules, and the like.
Those skilled in the art will appreciate that the data transfer device configuration shown in fig. 4 does not constitute a limitation of the data transfer device and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.
As shown in fig. 4, a memory 4005, which is a kind of computer storage medium, may include therein an operating system, a network communication module, a user interface module, and a data transmission program. Among them, the operating system is a program that manages and controls the hardware and software resources of the data transfer device, supporting the operation of the data transfer program as well as other software or programs.
In the data transmission device shown in fig. 4, the user interface 4003 is mainly used for connecting a terminal and performing data communication with the terminal, such as receiving user signaling data sent by the terminal; the network interface 4004 is mainly used for a background server and performs data communication with the background server; the processor 4001 may be configured to invoke a data transfer program stored in the memory 4005 and perform the steps of the data transfer method as described above.
The specific implementation of the data transmission device of the present application is substantially the same as the embodiments of the data transmission method, and is not described herein again.
In addition, an embodiment of the present application further provides a computer-readable storage medium, where a data transmission program is stored, and when the data transmission program is executed by a processor, the data transmission program implements the steps of the data transmission method described above.
The specific implementation of the computer-readable storage medium of the present application is substantially the same as the embodiments of the data transmission method, and is not described herein again.
In addition, to achieve the above object, the present application also provides a computer program product comprising: computer program which, when being executed by a processor, carries out the steps of the data transmission method as described above.
The specific implementation of the computer program product of the present application is substantially the same as the embodiments of the data transmission method, and is not described herein again.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The above-mentioned serial numbers of the embodiments of the present application are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solution of the present application may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes: instructions are used to cause a terminal device (which may be a mobile phone, a computer, a server, a device, or a network device, etc.) to perform the method described in the embodiments of the present application.
The above description is only a preferred embodiment of the present application, and not intended to limit the scope of the present application, and all modifications of equivalent structures and equivalent processes, which are made by the contents of the specification and the drawings of the present application, or which are directly or indirectly applied to other related technical fields, are included in the scope of the present application.
Claims (10)
1. A method of data transmission, the method comprising:
in the process of transmitting data to be transmitted, after the data content of a data fragment is transmitted to a target file, acquiring the distribution condition of illegal characters of the content stored in the target file; the data to be transmitted is segmented into a plurality of data segments; the storage content corresponds to the data content;
and if the distribution condition meets a preset data damage condition, determining that the stored content is damaged, and retransmitting the data content of the data fragment to the target file.
2. The method of claim 1, wherein before obtaining the distribution of illegal characters of the content stored in the target file, the method comprises:
polling a cache pipeline of the data transmission assembly to obtain a polling result; the data transmission component is used for transmitting data to be transmitted;
if the polling result is that cache data exists in the cache pipeline, transmitting the data content of one data fragment in the cache data to a target file; the cache data comprises at least one data slice.
3. The method of claim 2, wherein polling a cache pipe of a data transfer component to obtain a polling result comprises:
acquiring the data category of the data to be transmitted;
determining a number of polls to poll the data transmission component based on the data category;
and polling the cache pipeline of the data transmission assembly based on the polling times to obtain a polling result.
4. The method of claim 2, wherein if the polling result indicates that cache data exists in the cache pipe, transmitting data content of a data fragment in the cache data to a target file comprises:
if the polling result is that cache data exists in the cache pipeline, determining whether a target file exists in a data receiver;
if yes, determining whether an available data block exists in the target file;
and if so, writing the data content of one data fragment in the cache data into the available data block.
5. The method of claim 4, wherein if so, writing the data content of a data slice in the cache data to the available data block after writing the data content of the data slice to the available data block comprises:
starting a file input stream; the file input stream is used for inputting the storage content in the available data blocks;
if the file input stream fails to be started, deleting the storage content corresponding to the data fragments in the available data blocks, and returning the polling times of the cache pipeline to the polling times when the polling is started;
and retransmitting the data content corresponding to the deleted storage content to the available data block.
6. The method of claim 4, wherein after obtaining the distribution of illegal characters of the content stored in the target file, the method further comprises:
if the distribution condition does not meet the preset data damage condition, closing the file write stream;
if the closing of the file write stream fails, deleting the storage content corresponding to the data fragment in the available data block, and returning the polling times of the cache pipeline to the polling times when the polling is started;
and retransmitting the data content corresponding to the deleted storage content to the available data block.
7. The method of claim 2, wherein after determining that the storage content is corrupted if the distribution satisfies a predetermined data corruption condition and retransmitting the data content of the data slice to the target file, the method further comprises:
and if the polling times of polling the cache pipeline are equal to the preset polling times or the cache data are not polled from the cache pipeline, closing the daemon agent for guarding the data transmission process in the current round.
8. A data transmission apparatus, characterized in that the apparatus comprises:
the acquisition module is used for acquiring the distribution condition of illegal characters of the content stored in a target file after the data content of a data fragment is transmitted to the target file in the process of transmitting the data to be transmitted; the data to be transmitted is segmented into a plurality of data segments; the storage content corresponds to the data content;
and the determining module is used for determining that the storage content is damaged if the distribution condition meets a preset data damage condition, and retransmitting the data content of the data fragment to the target file.
9. A data transmission device, characterized in that the data transmission device comprises a memory, a processor and a data transmission program stored on the memory and executable on the processor, which data transmission program, when executed by the processor, implements the steps of the data transmission method as claimed in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that a data transmission program is stored thereon, which when executed by a processor implements the steps of the data transmission method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210761802.9A CN115085869A (en) | 2022-06-30 | 2022-06-30 | Data transmission method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210761802.9A CN115085869A (en) | 2022-06-30 | 2022-06-30 | Data transmission method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115085869A true CN115085869A (en) | 2022-09-20 |
Family
ID=83258169
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210761802.9A Pending CN115085869A (en) | 2022-06-30 | 2022-06-30 | Data transmission method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115085869A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060245484A1 (en) * | 2005-04-29 | 2006-11-02 | Calvin John C | Encoded serial data bit error detector |
| CN110291732A (en) * | 2016-12-21 | 2019-09-27 | 诺基亚技术有限公司 | The autonomous retransmission of damage data in wireless network |
| CN111245899A (en) * | 2019-12-31 | 2020-06-05 | 航天信息股份有限公司广州航天软件分公司 | Method and system for processing illegal message in web service environment |
| CN114430353A (en) * | 2021-12-28 | 2022-05-03 | 浙江伟星实业发展股份有限公司 | Modbus protocol-based data acquisition method, system, device and medium |
| CN114567506A (en) * | 2022-03-09 | 2022-05-31 | 平安科技（深圳）有限公司 | File uploading method and device, computer equipment and storage medium |
| WO2022116088A1 (en) * | 2020-12-03 | 2022-06-09 | 深圳市大疆创新科技有限公司 | Firmware data processing method and apparatus |
-
2022
- 2022-06-30 CN CN202210761802.9A patent/CN115085869A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060245484A1 (en) * | 2005-04-29 | 2006-11-02 | Calvin John C | Encoded serial data bit error detector |
| CN110291732A (en) * | 2016-12-21 | 2019-09-27 | 诺基亚技术有限公司 | The autonomous retransmission of damage data in wireless network |
| CN111245899A (en) * | 2019-12-31 | 2020-06-05 | 航天信息股份有限公司广州航天软件分公司 | Method and system for processing illegal message in web service environment |
| WO2022116088A1 (en) * | 2020-12-03 | 2022-06-09 | 深圳市大疆创新科技有限公司 | Firmware data processing method and apparatus |
| CN114430353A (en) * | 2021-12-28 | 2022-05-03 | 浙江伟星实业发展股份有限公司 | Modbus protocol-based data acquisition method, system, device and medium |
| CN114567506A (en) * | 2022-03-09 | 2022-05-31 | 平安科技（深圳）有限公司 | File uploading method and device, computer equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110716895B (en) | Target data archiving method, device, computer equipment and medium | |
| CN111414334A (en) | File fragment uploading method, device, equipment and storage medium based on cloud technology | |
| CN107395209B (en) | Data compression method, data decompression method and equipment thereof | |
| US10171608B2 (en) | Method, apparatus and computer program for modifying messages in a communications network | |
| CN110347341B (en) | Data clearing method and control equipment | |
| CN111680008B (en) | Log processing method and system, readable storage medium and intelligent device | |
| CN111382123B (en) | File storage method, device, equipment and storage medium | |
| CN108874825B (en) | Abnormal data verification method and device | |
| CN111708673A (en) | Log data compression method, device, equipment and storage medium | |
| CN110069217B (en) | Data storage method and device | |
| US20110055179A1 (en) | System for and method of partial file hashing | |
| CN110019039B (en) | Metadata-separated container format | |
| CN110888791A (en) | Log processing method, device, equipment and storage medium | |
| CN112969198A (en) | Data transmission method, terminal and storage medium | |
| CN112882743A (en) | Software upgrading method | |
| CN115085869A (en) | Data transmission method, device, equipment and storage medium | |
| CN112865927B (en) | Message delivery verification method, device, computer equipment and storage medium | |
| CN113868479A (en) | Method and device for processing service data | |
| CN109918231B (en) | Data reforming abnormity repairing method, device, equipment and storage medium | |
| CN114968963A (en) | File overwriting method and device and electronic equipment | |
| CN114116675A (en) | Data archiving method and device | |
| CN111866767B (en) | Message processing method, device, electronic equipment and readable medium | |
| CN111679835A (en) | Application processing method, device, terminal and storage medium | |
| CN111221556A (en) | Method and system for remotely upgrading RSU mirror image | |
| CN111695327A (en) | Method and device for repairing messy codes, electronic equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |