CN115314487A - Data storage method applied to reliable transmission and related device - Google Patents
Data storage method applied to reliable transmission and related device Download PDFInfo
- Publication number
- CN115314487A CN115314487A CN202210948594.3A CN202210948594A CN115314487A CN 115314487 A CN115314487 A CN 115314487A CN 202210948594 A CN202210948594 A CN 202210948594A CN 115314487 A CN115314487 A CN 115314487A
- Authority
- CN
- China
- Prior art keywords
- file
- message
- transmitted
- segments
- header
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/104—Peer-to-peer [P2P] networks
- H04L67/1074—Peer-to-peer [P2P] networks for supporting data block transmission mechanisms
- H04L67/1078—Resource delivery mechanisms
- H04L67/108—Resource delivery mechanisms characterised by resources being split in blocks or fragments
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Transfer Between Computers (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application provides a data storage method and a related device applied to reliable transmission, wherein the method comprises the following steps: receiving a message to be transmitted for reliable transmission; determining a file header of the message to be transmitted; carrying out segmentation processing on the message to be transmitted to obtain K file segments, wherein the K file segments have the same length, and determining record ending identifiers of the K file segments; constructing according to a preset file construction method according to the file header of the message to be transmitted, the K file fragments and the record ending marks of the K file fragments to obtain a target file; when the target file is stored or read, the number of times of reading and writing the file is reduced by storing or reading each message to be transmitted at most once, and the sequential reading and writing is convenient to realize batch reading and writing, so that the efficiency of storing or reading the file is improved.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a data storage method and a related apparatus for reliable transmission.
Background
When reliable transmission of messages is performed by using network communication, the messages need to be saved in files to prevent data loss after a program failure. At present, when a general message with a large length is stored, the service type and the length of a transmitted message cannot be determined in advance, the length of the message is large, the message is possible from a few bytes to a few megabytes, the message bytes cannot be stored in a single record of a database, and a file storage format capable of rapidly storing the variable length must be found.
At present, a plurality of file storage modes are generally adopted to solve the problem. For example, it can be implemented by using a fixed-length index file, plus a data file. In the index file, each record is of fixed length, and there is an offset in the fixed length record that points to where the data file is located. When the data is located, the record number is located in the index file, and then the data and other attribute information are read from the data file according to the offset pointing to the data file in the index file. This mode is used, for example, by the kafka et al message middleware.
The above method can solve the problem of file storage of reliable transmission, but the reading and writing of any record needs to be respectively read and written once in the index file and the data file, because two file reading and writing are involved and each message needs to be respectively read and written, the magnetic head of the magnetic disk is moved at least twice and is not sequentially read and written, and according to the common sense of computer storage, the magnetic disk efficiency is very low when the file is randomly read and written.
Disclosure of Invention
The embodiment of the application provides a data storage method and a related device applied to reliable transmission, and the method and the device can reduce the times of file reading and writing by storing or reading each message to be transmitted at most once, are sequential reading and writing and improve the efficiency of file storage or reading. In the embodiment of the application, the recorded line number is the record identifier for reliable transmission, and the use is simple and convenient.
A first aspect of an embodiment of the present application provides a data storage method applied to reliable transmission, where the method includes:
receiving a message to be transmitted for reliable transmission;
determining a file header of the message to be transmitted;
carrying out segmentation processing on the message to be transmitted to obtain K file segments, wherein the K file segments have the same length, and determining record ending identifiers of the K file segments;
constructing according to a preset file construction method according to the file header of the message to be transmitted, the K file fragments and the record ending marks of the K file fragments to obtain a target file;
and storing the target file.
In a possible implementation manner, the constructing according to the file header of the message to be transmitted, the K file segments, and the record end identifiers of the K file segments according to a preset file construction method to obtain the target file includes:
acquiring a file construction template corresponding to the message to be transmitted;
and constructing a file header of a message to be transmitted, K file segments and record ending marks of the K file segments according to the file construction template so as to obtain the target file.
In a possible implementation manner, the constructing according to the file header of the message to be transmitted, the K file segments, and the record end identifiers of the K file segments according to a preset file construction method to obtain the target file includes:
writing the file header into a file to obtain a first intermediate file;
sequentially writing recording end marks of K file segments of the first message to be transmitted, the effective lengths of the K file segments and the contents of the K file segments line by line in the first intermediate file to obtain a second intermediate file, and if the message is the last message, determining the second intermediate file as a target file;
and continuously repeating the processes, sequentially writing the subsequent messages to obtain a subsequent intermediate file, and if the message is the last message, determining the subsequent intermediate file as the target file.
In a possible implementation manner, the segmenting the message to be transmitted to obtain K file segments includes:
acquiring a preset single-line recording length;
judging whether the data volume of the message to be transmitted can be divided by the data volume corresponding to the preset single-line recording length, and if so, carrying out equal division on the message to be transmitted to obtain K file segments;
and if the data bits of the messages to be transmitted cannot be evenly divided, equally dividing the messages to be transmitted according to the preset single-line record length to obtain K-1 file fragments, and fixedly filling data bits of the messages not to be transmitted in the K-th file fragment to obtain the filled K-th file fragment.
In one possible implementation, the method further includes:
reading a message set to be transmitted;
the reading the set of messages to be transmitted includes:
and acquiring a file header of the message to be transmitted, wherein the file header comprises information of the length of the file header and a single-row recording length. The recorded line number is the record identification for reliable transmission;
reading M rows of data into a memory at one time according to the single row record length information in the file header, disassembling and recovering K message packets to be transmitted in the memory according to a record end mark, and merging the tail data of the time and the head data row of the next read memory to obtain a complete message packet to be transmitted if the record end mark of the last row of the read data is not ended;
and repeating the processes to obtain M message packages to be transmitted, wherein M is a positive integer greater than or equal to K.
A second aspect of an embodiment of the present application provides a data storage device applied to reliable transmission, where the device includes:
a receiving unit, configured to receive a message to be transmitted for reliable transmission;
the determining unit is used for determining the file header of the message to be transmitted;
the processing unit is used for carrying out segmentation processing on the message to be transmitted so as to obtain K file segments, wherein the lengths of the K file segments are the same, and the recording end identifiers of the K file segments are determined;
the construction unit is used for constructing according to a file header of the message to be transmitted, the K file segments and the recording end marks of the K file segments by a preset file construction method so as to obtain a target file;
and the storage unit is used for storing the target file.
In one possible implementation, the construction unit is configured to:
acquiring a file construction template corresponding to the message to be transmitted;
and constructing a file header of a message to be transmitted, K file segments and record ending marks of the K file segments according to the file construction template so as to obtain the target file.
In one possible implementation, the construction unit is configured to:
writing the file header in a file to obtain a first intermediate file;
and sequentially writing the record ending marks of the K file segments of the first message to be transmitted, the effective lengths of the K file segments and the contents of the K file segments line by line in the first intermediate file to obtain the second intermediate file. If the message is the last message, the message is the target file.
And continuously repeating the processes, and sequentially writing the subsequent messages to obtain the subsequent intermediate files. If the message is the last message, the message is the target file.
In a possible implementation manner, in the aspect of segmenting the message to be transmitted to obtain K file segments, the processing unit is configured to:
acquiring a preset single-line recording length;
judging whether the data volume of the message to be transmitted can be evenly divided by the data volume corresponding to the preset single-line recording length, and if so, carrying out even division on the message to be transmitted to obtain K file segments;
and if the data bits of the messages to be transmitted cannot be evenly divided, equally dividing the messages to be transmitted according to the preset single-line record length to obtain K-1 file fragments, and fixedly filling data bits of the messages not to be transmitted in the K-th file fragment to obtain the filled K-th file fragment.
In one possible implementation, the apparatus is further configured to:
reading a message set to be transmitted;
the reading the set of messages to be transmitted includes:
acquiring a file header of the message to be transmitted, wherein the file header comprises information of a file header length and a single-row recording length;
according to the length information of the single-row record in the file header, reading M rows of data into a memory at one time, resolving and recovering to obtain K message packages to be transmitted in the memory according to a record end mark, and if the record end mark of the last row of the read data is not ended, merging the tail data of the time with the head data of the next read memory to obtain a complete message package to be transmitted;
and repeating the processes to obtain M message packets to be transmitted, wherein M is a positive integer greater than or equal to K.
A third aspect of embodiments of the present application provides a terminal, including a processor, an input device, an output device, and a memory, where the processor, the input device, the output device, and the memory are connected to each other, where the memory is used to store a computer program, and the computer program includes program instructions, and the processor is configured to call the program instructions to execute the step instructions in the first aspect of embodiments of the present application.
A fourth aspect of embodiments of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for electronic data exchange, wherein the computer program causes a computer to perform some or all of the steps as described in the first aspect of embodiments of the present application.
A fifth aspect of embodiments of the present application provides a computer program product, wherein the computer program product comprises a non-transitory computer readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps as described in the first aspect of embodiments of the present application. The computer program product may be a software installation package.
The embodiment of the application has at least the following beneficial effects:
the method comprises the steps of receiving a message to be transmitted for reliable transmission, determining a file header of the message to be transmitted, carrying out segmentation processing on the message to be transmitted to obtain K file segments, wherein the K file segments are the same in length, determining recording end marks of the K file segments, and constructing according to the file header of the message to be transmitted, the K file segments and the recording end marks of the K file segments by a preset file construction method to obtain a target file; the target file is stored, so that the target file constructed by the message to be transmitted is stored once and is read and written in a sequence with higher efficiency, the times of file storage can be reduced, and the efficiency of file storage is improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1A is a schematic flowchart of a data storage method applied to reliable transmission according to an embodiment of the present application;
FIG. 1B is a schematic diagram of an object file according to an embodiment of the present application
FIG. 2 is a flow chart illustrating another data storage method applied to reliable transmission according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of a server according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of a data storage device applied to reliable transmission according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The terms "first," "second," and the like in the description and claims of the present application and in the above-described drawings are used for distinguishing between different objects and not for describing a particular order. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.
Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the specification. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.
In order to better understand the data storage method applied to reliable transmission provided by the embodiments of the present application, a brief description is first given below to a scenario of the data storage method applied to reliable transmission. When reliable transmission of messages is performed by using network communication, the messages need to be saved in files to prevent the problem of data loss caused by program failure. Two types are distinguished:
(1) For messages of a small or fixed length variation in length.
Messages of very small size variations usually have a well-defined business meaning. In general, the service field corresponding to each message is stored in a database in a form of record to support reliable transmission.
(2) For general purpose messages of widely varying lengths.
For a message (data to be transmitted) in software similar to message middleware, the service type and the length of the transmitted message cannot be determined in advance, the length of the message is greatly changed, the message can be stored in a single record of a database from a few bytes to a few megabytes, and the problem is generally solved by a plurality of file storage modes when the general message with the greatly changed length is stored at present. For example, it can be implemented using a fixed-length index file, plus a data file. In the index file, each record is of fixed length, and there is an offset in the fixed length record that points to where the data file is located. When the data is located, the record number is located in the index file, and then the data and other attribute information are read from the data file according to the offset pointing to the data file in the index file. This mode is used, for example, by the kafka et al message middleware.
The method can solve the problem of file storage of reliable transmission, but the reading and writing of any record needs to be performed in the index file and the data file once respectively, although each file reading and writing is sequential reading and writing, because two file reading and writing are involved, and each message needs to be read respectively, batch reading cannot be performed, and the moving times of the magnetic head are increased (even if a lot of data is read to a memory once, each record needs to be split relatively complexly, and mapping calculation is performed with the file offset). The reading and writing of the magnetic disk are very time-consuming operations relative to the reading of the memory, and particularly, the moving of the magnetic head of the magnetic disk from the index file to the positioning of the data file is random positioning, so that a large amount of time is consumed, and the efficiency of data storage or reading is low.
Referring to fig. 1A, fig. 1A is a schematic flow chart illustrating a data storage method for reliable transmission according to an embodiment of the present disclosure. As shown in fig. 1A, the method may be applied to a server, and the method includes:
101. a message to be transmitted for reliable transmission is received.
When the file is transmitted reliably, the message to be transmitted may be a message to be transmitted sent from the client, may also be a message to be transmitted sent from another server, and certainly may also be a message to be transmitted for reliable transmission sent by another device.
102. And determining the file header of the message to be transmitted.
The version number and the like of the message to be transmitted can be acquired, the length of the file header, the preset single-line recording length and the like can be acquired, and the length of the file header and the preset single-line recording length can be set through historical data or empirical values. Therefore, the file header can be determined according to the version number of the message to be transmitted, the length of the file header, the preset single-line recording length and the like.
103. And carrying out segmentation processing on the message to be transmitted to obtain K file fragments, wherein the K file fragments have the same length, and determining the record ending marks of the K file fragments.
The recording sections are all "no" except that the "recording end flag" of the last section is "yes".
When the message to be transmitted is processed in a segmented manner, the message to be transmitted may be processed in a segmented manner according to a length corresponding to a preset single-line record length, so as to obtain K file segments. And determining an end-of-recording flag of the file segment, where the end-of-recording flag is used to characterize whether the file segment is the last file segment of the message to be transmitted, for example, the end-of-recording flag is 0 or 1, where 1 indicates that the recording is not ended, and 0 indicates that the recording is ended to this segment (i.e., the last segment of the message to be transmitted). Binary indicates that the length is fixed to 1 byte. The actual length of the last file segment of the K file segments may be smaller than the length of the file segment, and the actual length of the last segment may be understood as the length of the existing data, since the filled content may exist in the last segment (when the length of the message to be transmitted is not evenly divisible by the length of the file segment).
104. And constructing according to the file header of the message to be transmitted, the K file fragments and the recording end marks of the K file fragments according to a preset file construction method to obtain a target file.
The file construction template corresponding to the message to be transmitted can be obtained, and construction is performed based on the construction template to obtain the target file. Or obtaining the target file by writing the intermediate file into the memory.
After the target file is formed, subsequent messages to be transmitted can be written into the target file, so that the plurality of messages to be transmitted can be stored through one file.
105. And storing the target file.
The target file can be stored and written in an additional mode at one time, or the target file determined by a plurality of messages to be transmitted can be cached, and the target file can be written in one time after the target file is cached to a certain number of records or exceeds a certain time interval. Because the message middleware such as Kafka and the like needs to write the index file and the data file at least once respectively, compared with the traditional message storage mode, the method has the advantages that the writing times are reduced, and the storage efficiency is greatly improved.
If other messages to be transmitted are received subsequently, the messages can be processed according to the method for determining the target file, the files determined by the messages are added to the target file, the files can be added according to the time sequence during the adding, the file header also needs to be updated during the adding, and the related information of the subsequent messages is added to the file header.
In this example, a message to be transmitted for reliable transmission is received, a file header of the message to be transmitted is determined, the message to be transmitted is subjected to segmentation processing to obtain K file segments, the lengths of the K file segments are the same, recording end identifiers of the K file segments are determined, and a target file is obtained by constructing the file header of the message to be transmitted, the K file segments and the recording end identifiers of the K file segments according to a preset file construction method; the target file is stored, so that the target file constructed by the message to be transmitted is stored once and is read and written in a sequence with higher efficiency, the times of file storage can be reduced, and the efficiency of file storage is improved.
In a possible implementation manner, a possible method for constructing according to the file header of the message to be transmitted, the K file fragments, and the record ending identifiers of the K file fragments by using a preset file construction method to obtain a target file includes:
a1, acquiring a file construction template corresponding to the message to be transmitted;
and A2, constructing a file header of a message to be transmitted, K file fragments and record ending marks of the K file fragments according to the file construction template to obtain the target file.
Wherein the file construction template can be determined according to the type of the message to be transmitted. Different types correspond to different file construction templates, and the type of the message to be transmitted can comprise a fixed length type and a non-fixed length type. The fixed length may be understood as that the data sizes of a plurality of messages to be transmitted are the same, or the difference between the data sizes is small, for example, the size difference is smaller than a fixed threshold. A non-fixed length type may be understood as a type where the difference between the data sizes of multiple messages to be transmitted is large, e.g. the gap size is larger than a fixed threshold. The fixed threshold is set by empirical values or historical data.
The file header, the K file segments and the K file segments of the message to be transmitted can be constructed according to the construction method shown by the file construction template, so that the target file is obtained.
In this example, the file can be quickly constructed through the file construction template to obtain the target file, so that the efficiency of determining the target file is improved.
In a possible implementation manner, another possible method for constructing according to a file header of the message to be transmitted, K file fragments, and record end identifiers of the K file fragments by a preset file construction method to obtain a target file includes:
b1, writing the file header in a file to obtain a first intermediate file;
and B2, sequentially writing recording end marks of K file segments of the first message to be transmitted, the effective lengths of the K file segments and the contents of the K file segments line by line in the first intermediate file to obtain a second intermediate file, and if the message is the last message, determining that the second intermediate file is a target file.
And B3, continuously repeating the processes, sequentially writing the subsequent messages to obtain a subsequent intermediate file, and if the message is the last message, determining the subsequent intermediate file as the target file.
Wherein the file header is written first. The file header may contain information such as the version number of the file, the length of the file header, the length of each line record, etc., and may be defined by the user himself, so as to obtain the first intermediate file.
The data to be written to the file is organized in memory. For each file fragment, the following loop is performed:
the recording end flag is written first. If it is the last file fragment of the present service message, a 0 is written at the flag, otherwise a1 is written.
And then writing the effective length of the data of the segment of the line. If the fragment is the last fragment of the business message, the effective length of the actual fragment data is written in the field, otherwise, the fragment fixed length is written. The effective length can be understood as the length of the file segment in which data is stored. If not the last segment, the effective length of the data may be understood as the length of the segment.
As shown in FIG. 1B, FIG. 1B is a schematic diagram of an object file. The clip data in fig. 1B corresponds to a file clip, the recording end flag corresponds to a recording end flag, and the effective clip length corresponds to the effective length of the file clip.
In a possible implementation manner, a possible method for performing a segmentation process on the message to be transmitted to obtain K file segments includes:
c1, acquiring a preset single-row record length;
c2, judging whether the data volume of the message to be transmitted can be divided by the data volume corresponding to the preset single-line recording length, and if so, equally dividing the message to be transmitted to obtain the K file segments;
and C3, if the file fragments cannot be divided completely, equally dividing the message to be transmitted according to the preset single-line record length to obtain K-1 file fragments, and fixedly filling data bits of the message not to be transmitted in the K file fragment to obtain the filled K file fragment.
Wherein the preset single-line record length can be set by empirical values or historical data.
When the file segment cannot be divided completely, the length of the first K-1 file segments is the length specified by the preset single-row record length, the data in the K-1 file segments is the data to be transmitted, part of the K-th file segment is the data to be transmitted, and part of the K-th file segment is padding data for fixed padding, and the padding data can be '0', for example. The location of the padding data may be a location after the data to be transmitted, but may of course be other locations.
In a possible implementation manner, the message to be transmitted may also be read, and the specific reading method is as follows:
and reading a message set to be transmitted. The method for specifically reading the message set to be transmitted includes:
and D1, acquiring a file header of the message to be transmitted, wherein the file header comprises information of the length of the file header and a single-row recording length. The recorded row number is the record identification for reliable transmission;
d2, reading M rows (the value of M can be 100-10000, and is specifically determined by the size of the memory) of data into the memory at one time according to the single-row record length information in the file header, disassembling and recovering to obtain K message packets to be transmitted in the memory according to a record ending mark, and merging the tail data of this time with the head data row of the next read memory to obtain a complete message packet to be transmitted if the record ending mark of the last row of the read data of this time is not ended;
and D3, repeating the processes to obtain M message packages to be transmitted, wherein M is a positive integer greater than or equal to K.
When data is read, a plurality of messages to be transmitted (namely a message set to be transmitted) can be read at one time, and during reading, a file header can be read in storage firstly, wherein the file header comprises information of the length of the file header and the length of a single-row record. For example, if the length corresponding to the preset single-line recording length is 500 bytes, the read message to be transmitted may be 20 ten thousand lines, each line includes a file segment, a recording end identifier, and an effective length of the file segment, so that the size of data that can be read each time is 10 MB. In particular, when data reading is performed, sequential reading and writing can be realized.
The data storage method of the embodiment of the application can (1) reduce the file reading and writing times and realize sequential reading and writing, thereby improving the efficiency of reading and writing the file. Especially for reading files, the disk operation that each record needs to be read twice before can be realized, and the disk operation that each file record is read once every 20 ten thousand lines can be realized, thereby greatly improving the disk reading performance. ( Here, the record length is 500 bytes, so that 20 ten thousand lines of file records can be read at a time. If the parameters are different, the number of times of reading the disk is different )
(2) The storage of all information is realized by one file, so that the moving times of the magnetic head are reduced, the storage performance is improved, the interface is simpler, and the file sorting and moving are more brief.
(3) The method separates the two concepts of the length of the message packet of the service layer and the maximum packet length which can be sent each time by reliable transmission, embodies the design idea of layering the service layer and the transmission layer, and simultaneously, the design can support the scenes that the transmission layer has length limitation for each time of sending data (such as UDP transmission, a message middleware system with the maximum length limitation for a single packet, and the like), and has wider adaptability.
Referring to fig. 2, fig. 2 is a schematic flowchart illustrating a data storage method for reliable transmission according to an embodiment of the present disclosure. As shown in fig. 2, the method may be applied to a server, and the method includes:
201. receiving a message to be transmitted for reliable transmission;
202. determining a file header of the message to be transmitted;
203. carrying out segmentation processing on the message to be transmitted to obtain K file segments, wherein the K file segments have the same length, and determining record ending identifiers of the K file segments;
204. constructing according to a preset file construction method according to the file header of the message to be transmitted, the K file fragments and the record ending marks of the K file fragments to obtain a target file;
205. storing the target file;
206. and acquiring a file header of the message to be transmitted, wherein the file header comprises information of the length of the file header and a single-row recording length. The recorded row number is the record identification for reliable transmission;
207. reading M rows of data into a memory at one time according to the single row record length information in the file header, disassembling and recovering K message packets to be transmitted in the memory according to a record end mark, and merging the tail data of the time and the head data row of the next read memory to obtain a complete message packet to be transmitted if the record end mark of the last row of the read data is not ended;
208. and repeating the processes to obtain M message packets to be transmitted, wherein M is a positive integer greater than or equal to K.
In this example, a plurality of messages to be transmitted can be read at one time, and compared with the existing scheme, when only data corresponding to one message to be transmitted can be read each time, the time for reading data can be greatly reduced, and the reading efficiency is improved.
In accordance with the foregoing embodiments, please refer to fig. 3, fig. 3 is a schematic structural diagram of a server according to an embodiment of the present application, and as shown in the drawing, the server includes a processor, an input device, an output device, and a memory, where the processor, the input device, the output device, and the memory are connected to each other, where the memory is used to store a computer program, the computer program includes program instructions, the processor is configured to call the program instructions, and the program includes instructions for executing the following steps;
receiving a message to be transmitted for reliable transmission;
determining a file header of the message to be transmitted;
carrying out segmentation processing on the message to be transmitted to obtain K file segments, wherein the K file segments have the same length, and determining record ending identifiers of the K file segments;
constructing according to a preset file construction method according to the file header of the message to be transmitted, the K file fragments and the record ending marks of the K file fragments to obtain a target file;
and storing the target file.
The above description has introduced the solution of the embodiment of the present application mainly from the perspective of the method-side implementation process. It is understood that the terminal includes corresponding hardware structures and/or software modules for performing the respective functions in order to implement the above-described functions. Those of skill in the art will readily appreciate that the present application is capable of hardware or a combination of hardware and computer software implementing the various illustrative elements and algorithm steps described in connection with the embodiments provided herein. Whether a function is performed in hardware or computer software drives hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiment of the present application, the terminal may be divided into the functional units according to the method example, for example, each functional unit may be divided corresponding to each function, or two or more functions may be integrated into one processing unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit. It should be noted that the division of the unit in the embodiment of the present application is schematic, and is only a logic function division, and there may be another division manner in actual implementation.
In accordance with the above, please refer to fig. 4, fig. 4 is a schematic structural diagram of a data storage device for reliable transmission according to an embodiment of the present application. As shown in fig. 4, the apparatus includes:
a receiving unit 401, configured to receive a message to be transmitted for reliable transmission;
a determining unit 402, configured to determine a file header of the message to be transmitted;
a processing unit 403, configured to perform segmentation processing on the message to be transmitted to obtain K file segments, where the K file segments have the same length, and determine record end identifiers of the K file segments;
a constructing unit 404, configured to construct according to a preset file constructing method, according to the file header of the message to be transmitted, the K file segments, and the record end identifiers of the K file segments, so as to obtain a target file;
a storage unit 405, configured to store the target file.
In one possible implementation manner, the constructing unit 404 is configured to:
acquiring a file construction template corresponding to the message to be transmitted;
and constructing a file header, K file fragments and record ending marks of the K file fragments of the message to be transmitted according to the file construction template to obtain the target file.
In one possible implementation manner, the building unit 404 is configured to:
writing the file header into a file to obtain a first intermediate file;
sequentially writing recording end marks of K file segments of the first message to be transmitted, the effective lengths of the K file segments and the contents of the K file segments line by line in the first intermediate file to obtain a second intermediate file, and if the message is the last message, determining the second intermediate file to be a target file;
and continuously repeating the processes, sequentially writing the subsequent messages to obtain a subsequent intermediate file, and if the message is the last message, determining the subsequent intermediate file as the target file.
In a possible implementation manner, in the aspect of segmenting the message to be transmitted to obtain K file segments, the processing unit 403 is configured to:
acquiring a preset single-line recording length;
judging whether the data volume of the message to be transmitted can be evenly divided by the data volume corresponding to the preset single-line recording length, and if so, carrying out even division on the message to be transmitted to obtain K file segments;
and if the file fragments cannot be evenly divided, equally dividing the message to be transmitted according to the preset single-row record length to obtain K-1 file fragments, and fixedly filling data bits of the message not to be transmitted in the K file fragment to obtain the filled K file fragment.
In one possible implementation, the apparatus is further configured to:
reading a message set to be transmitted;
the reading the set of messages to be transmitted includes:
and acquiring a file header of the message to be transmitted, wherein the file header comprises information of the length of the file header and a single-row recording length. The recorded row number is the record identification for reliable transmission;
reading M rows of data into a memory at one time according to the single row record length information in the file header, disassembling and recovering K message packets to be transmitted in the memory according to a record end mark, and merging the tail data of the time and the head data row of the next read memory to obtain a complete message packet to be transmitted if the record end mark of the last row of the read data is not ended;
and repeating the processes to obtain M message packages to be transmitted, wherein M is a positive integer greater than or equal to K.
Embodiments of the present application also provide a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, and the computer program enables a computer to execute part or all of the steps of any one of the data storage methods applied to reliable transmission as described in the above method embodiments.
Embodiments of the present application also provide a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program causes a computer to execute part or all of the steps of any one of the data storage methods applied to reliable transmission as described in the above method embodiments.
It should be noted that, for simplicity of description, the above-mentioned method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present application is not limited by the order of acts described, as some steps may occur in other orders or concurrently depending on the application. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required in this application.
In the foregoing embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus may be implemented in other manners. For example, the above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one type of division of logical functions, and there may be other divisions when actually implementing, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not implemented. In addition, the shown or discussed coupling or direct coupling or communication connection between each other may be through some interfaces, indirect coupling or communication connection between devices or units, and may be in an electrical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit may be implemented in the form of hardware, or may be implemented in the form of a software program module.
The integrated unit, if implemented in the form of a software program module and sold or used as a stand-alone product, may be stored in a computer readable memory. Based on such understanding, the technical solution of the present application may be substantially implemented or a part of or all or part of the technical solution contributing to the prior art may be embodied in the form of a software product stored in a memory, and including several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method described in the embodiments of the present application. And the aforementioned memory comprises: various media capable of storing program codes, such as a usb disk, a read-only memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and the like.
Those skilled in the art will appreciate that all or part of the steps of the methods of the above embodiments may be implemented by a program, which is stored in a computer-readable memory, the memory including: flash memory disks, read-only memory, random access memory, magnetic or optical disks, and the like.
The foregoing detailed description of the embodiments of the present application has been presented to illustrate the principles and implementations of the present application, and the above description of the embodiments is only provided to help understand the method and the core concept of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.
Claims (10)
1. A data storage method for reliable transmission, the method comprising:
receiving a message to be transmitted for reliable transmission;
determining a file header of the message to be transmitted;
carrying out segmentation processing on the message to be transmitted to obtain K file segments, wherein the K file segments have the same length, and determining record ending identifiers of the K file segments;
constructing according to a preset file construction method according to the file header of the message to be transmitted, the K file fragments and the record ending marks of the K file fragments to obtain a target file;
and storing the target file.
2. The method according to claim 1, wherein the constructing according to the file header of the message to be transmitted, the K file fragments and the record end identifiers of the K file fragments according to a preset file construction method to obtain the target file comprises:
acquiring a file construction template corresponding to the message to be transmitted;
and constructing a file header of a message to be transmitted, K file segments and record ending marks of the K file segments according to the file construction template so as to obtain the target file.
3. The method according to claim 1, wherein the constructing according to the file header of the message to be transmitted, the K file fragments, and the record ending identifiers of the K file fragments is performed according to a preset file construction method to obtain a target file, comprises:
writing the file header in a file to obtain a first intermediate file;
sequentially writing recording end marks of K file segments of the first message to be transmitted, the effective lengths of the K file segments and the contents of the K file segments line by line in the first intermediate file to obtain a second intermediate file, and if the message is the last message, determining the second intermediate file as a target file;
and continuously repeating the processes, sequentially writing the subsequent messages to obtain a subsequent intermediate file, and if the message is the last message, determining the subsequent intermediate file as the target file.
4. The method according to any one of claims 1 to 3, wherein the segmenting the message to be transmitted to obtain K file segments comprises:
acquiring a preset single-line recording length;
judging whether the data volume of the message to be transmitted can be evenly divided by the data volume corresponding to the preset single-line recording length, and if so, carrying out even division on the message to be transmitted to obtain K file segments;
and if the data bits of the messages to be transmitted cannot be evenly divided, equally dividing the messages to be transmitted according to the preset single-line record length to obtain K-1 file fragments, and fixedly filling data bits of the messages not to be transmitted in the K-th file fragment to obtain the filled K-th file fragment.
5. The method of claim 4, further comprising:
reading a message set to be transmitted;
the reading the set of messages to be transmitted includes:
acquiring a file header of the message to be transmitted, wherein the file header comprises information of a file header length and a single-row recording length;
according to the length information of the single-row record in the file header, reading M rows of data into a memory at one time, resolving and recovering to obtain K message packages to be transmitted in the memory according to a record end mark, and if the record end mark of the last row of the read data is not ended, merging the tail data of the time with the head data of the next read memory to obtain a complete message package to be transmitted;
and repeating the processes to obtain M message packets to be transmitted, wherein M is a positive integer greater than or equal to K.
6. A data storage device for reliable transmission, the device comprising:
a receiving unit, configured to receive a message to be transmitted for reliable transmission;
the determining unit is used for determining the file header of the message to be transmitted;
the processing unit is used for carrying out segmentation processing on the message to be transmitted so as to obtain K file segments, wherein the lengths of the K file segments are the same, and the recording end identifiers of the K file segments are determined;
the construction unit is used for constructing according to a file header of the message to be transmitted, the K file segments and the recording end marks of the K file segments by a preset file construction method so as to obtain a target file;
and the storage unit is used for storing the target file.
7. The apparatus of claim 6, wherein the construction unit is configured to:
acquiring a file construction template corresponding to the message to be transmitted;
and constructing a file header, K file fragments and record ending marks of the K file fragments of the message to be transmitted according to the file construction template to obtain the target file.
8. The apparatus of claim 6, wherein the construction unit is configured to:
writing the file header in a file to obtain a first intermediate file;
sequentially writing recording end marks of K file segments of the first message to be transmitted, the effective lengths of the K file segments and the contents of the K file segments line by line in the first intermediate file to obtain a second intermediate file, and if the message is the last message, determining the second intermediate file as a target file;
and continuously repeating the processes, sequentially writing the subsequent messages to obtain a subsequent intermediate file, and if the message is the last message, determining the subsequent intermediate file as the target file.
9. A terminal, characterized in that it comprises a processor, an input device, an output device and a memory, said processor, input device, output device and memory being interconnected, wherein said memory is used to store a computer program comprising program instructions, said processor being configured to invoke said program instructions to perform the method according to any of the claims 1-5.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program comprising program instructions that, when executed by a processor, cause the processor to carry out the method according to any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210948594.3A CN115314487A (en) | 2022-08-09 | 2022-08-09 | Data storage method applied to reliable transmission and related device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210948594.3A CN115314487A (en) | 2022-08-09 | 2022-08-09 | Data storage method applied to reliable transmission and related device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115314487A true CN115314487A (en) | 2022-11-08 |
Family
ID=83861236
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210948594.3A Pending CN115314487A (en) | 2022-08-09 | 2022-08-09 | Data storage method applied to reliable transmission and related device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115314487A (en) |
-
2022
- 2022-08-09 CN CN202210948594.3A patent/CN115314487A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107911249B (en) | Method, device and equipment for sending command line of network equipment | |
| CN106874348B (en) | File storage and index method and device and file reading method | |
| CN102768662B (en) | A kind of method and apparatus Loaded Image | |
| US8422786B2 (en) | Analyzing documents using stored templates | |
| CN104038450A (en) | Message transmission method and apparatus based on PCIE bus | |
| CN111008183B (en) | Storage method and system for business wind control log data | |
| CN108230487A (en) | The method and apparatus of shared camera resource | |
| CN106293542B (en) | Method and device for decompressing file | |
| CN115314487A (en) | Data storage method applied to reliable transmission and related device | |
| CN108073709B (en) | Data recording operation method, device, equipment and storage medium | |
| CN111367916A (en) | Data storage method and device | |
| CN104506390A (en) | Log storage method and device of road test system | |
| CN115033544A (en) | Data compression method, device, equipment and medium based on relation numerical value | |
| CN114238250A (en) | Program file compression method and device and electronic equipment | |
| CN102891732A (en) | Data sending method and device and data receiving method and device | |
| CN107689996B (en) | Data transmission method and device and terminal equipment | |
| US8375066B2 (en) | Generating unique identifiers | |
| US8788483B2 (en) | Method and apparatus for searching in a memory-efficient manner for at least one query data element | |
| CN111625502A (en) | Data reading method and device, storage medium and electronic device | |
| CN116708480B (en) | Data synchronization method based on Datax frame | |
| CN102542013B (en) | Realize the random access in the object in ZIP files | |
| CN114661772B (en) | Data processing method and related device | |
| CN111737658B (en) | Method, system, device and medium for limiting quota of ZooKeeper data node | |
| CN111193696B (en) | Data transmitting method, data receiving method, data transmitting device, data receiving device, computer equipment and storage medium | |
| CN115952133A (en) | Rich text data processing method, system and related equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |