CN115629711A - Packet writing alignment method, system, device and medium - Google Patents
Packet writing alignment method, system, device and medium Download PDFInfo
- Publication number
- CN115629711A CN115629711A CN202211348879.XA CN202211348879A CN115629711A CN 115629711 A CN115629711 A CN 115629711A CN 202211348879 A CN202211348879 A CN 202211348879A CN 115629711 A CN115629711 A CN 115629711A
- Authority
- CN
- China
- Prior art keywords
- packet
- batch
- brushing
- current
- judging whether
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/062—Securing storage systems
- G06F3/0622—Securing storage systems in relation to access
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The application discloses a packet writing alignment method, a system, a device and a medium, which are applied to the field of data transmission, wherein a packet sent by a client is obtained, whether the current lower batch exceeds a lower batch threshold value after the packet is added into the current lower batch is judged, if the current lower batch exceeds the lower batch threshold value, the packet is logically segmented and marked as segmented, the part of the packet, which is smaller than the lower batch threshold value, is added into the current lower batch, and the rest part of the packet is added into the next lower batch; if not, marking the packet as not segmented, adding the packet into the current lower batch, judging whether the current lower batch added into the waiting lower batch meets the lower brushing condition or not, and if so, brushing the current lower batch. The method is divided according to the lower brushing threshold value, and the part which does not conform to the lower brushing threshold value is processed in batches, so that the speed of writing in a magnetic disk and reading data is improved, and the occurrence of data messy codes and errors is reduced.
Description
Technical Field
The present application relates to the field of data transmission, and in particular, to a method, a system, an apparatus, and a medium for packet write alignment.
Background
With the development of the technology, when data is written in, a distributed file System (HDFS) implemented by a distributed System infrastructure is connected with a data node, a client divides a storage and processing unit (Block) in a database into a plurality of data packets (packets) in communication transmission smaller than the Block, and then the data packets (packets) are sent to a server, wherein the packets include sequence numbers of the data packets, writing positions, writing data lengths and the like, and the data packets are used for the server to perform down-refresh and upgrade.
At present, when a client sends a packet to a server, in many practical application scenarios, writing positions in the packet are not written in an aligned manner, which may cause that when writing into a disk, the same packet may be written into multiple storage objects, which may cause an increase in time consumed for writing into the disk during addressing, and even cause a situation of messy codes, and similarly, when reading data, the time for reading may also be increased, which may also cause an error.
How to write data into a disk after aligning packets is an urgent problem to be solved by those skilled in the art when writing data through HDFS.
Disclosure of Invention
The application aims to provide a packet writing alignment method, which can solve the problem that when data are written through an HDFS (Hadoop distributed File System), a packet is not aligned, brushed down and written into a disk.
In order to solve the above technical problem, the present application provides a packet writing alignment method, including:
acquiring a packet sent by a client;
judging whether the current lower brushing batch exceeds a lower brushing threshold value after the packet is added into the current lower brushing batch;
if the current lower batch is not the lower batch, performing logic segmentation on the packet and marking the packet as segmented, adding the part of the packet, which is smaller than the lower brushing threshold value, into the current lower batch, and adding the rest part into the next lower batch;
if not, marking the packet as not segmented, and adding the packet into the current lower batch;
judging whether the current lower brushing batch meets a lower brushing condition;
and if so, adding the current batch refreshing into a queue waiting for refreshing.
Preferably, the logically segmenting the packet includes:
marking the part of the packet exceeding the lower brushing threshold as a sliced part.
Preferably, the judging whether the current brushing batch meets the brushing condition comprises:
acquiring the maximum packet size in the current lower batch;
judging whether the sequence number of the packet and the offset in the data block are both 0 or not;
if the number of the packets is 0, adding the packet packets into the queue waiting for lower brushing, and replacing the new lower brushing batch as the next lower brushing batch;
judging whether the packet is segmented or not;
if the packet is segmented, adding the part of the packet, which is smaller than the preset lower brushing threshold value, into the current lower brushing batch, adding the segmented part into the next lower brushing batch, and judging whether the data length of the packet is smaller than the maximum packet size in the lower brushing batch where the packet is located;
if the packet size is smaller than the preset size, adding the packet into the next lower brushing batch;
if not, adding the packet into the current lower batch;
and if the packet is not segmented, adding the packet into the current lower batch.
Preferably, after the determining that the sequence number of the packet and the offset in the data block are not both 0, the method further includes:
judging whether the data length of the packet is smaller than the maximum packet size in the lower brushing batch;
and if the number of the packets is smaller than the preset number, adding the packet packets into the queue waiting for brushing, and replacing the new lower brushing batch as the next lower brushing batch.
Preferably, after the determining that the data length of the packet is not smaller than the maximum packet size in the lower batch where the packet is located, the method further includes:
judging whether the packet is segmented or not;
if the packet is segmented, adding the part of the packet smaller than the preset lower brushing threshold value into the queue waiting for lower brushing, and adding the segmented part into the next lower brushing batch;
judging whether the packet is segmented or not;
if not, adding the packet into the current lower batch;
if so, adding the already cut portion to the next lower batch.
Preferably, after the determining that the packet is not split, the method further includes:
judging whether the current lower brushing batch where the packet is located exceeds the lower brushing threshold;
and if the current lower batch exceeds the lower brushing threshold, adding the part of the current lower batch exceeding the lower brushing threshold into the queue waiting for lower brushing, adding the rest part into the next lower batch, and entering the step of judging whether the packet is segmented or not.
Preferably, after it is determined that the current lower brushing batch where the packet is located is smaller than the lower brushing threshold, the method further includes:
judging whether the packet is the last packet in the data block;
and if the packet is the last packet, adding the packet into the queue waiting for lower brushing, replacing the new lower brushing batch as the next lower brushing batch, and entering the step of judging whether the packet is segmented or not.
And if the packet is not the last packet, adding the packet into the current lower batch.
Preferably, after the adding the current brushing batch into the queue waiting for brushing, the method further comprises:
judging whether the packet is not the last packet;
if not, acquiring a first lower brushing batch from the waiting lower brushing queue, and judging whether the last packet in the first lower brushing batch is the last packet in the data block;
if the packet is the last packet, judging whether the first lower batch only has one packet;
if only one packet is available, adding the first lower batch to a confirmation collection to wait for the confirmation of the client, returning the confirmation to the client, removing the first lower batch from the waiting lower batch queue, and returning to the step of judging whether the packet is not the last packet;
if the number of the lower batch packets is more than one, adding the first lower batch packet into a confirmation collection to wait for the confirmation of the client, entering the step of removing the first lower batch packet from the waiting lower batch packet after the lower batch packet is finished, and returning to the step of judging whether the packet is not the last packet;
and if not, entering the step of adding the first batch down to a confirmation collection to wait for confirmation of the client, and after finishing the brushing of the first batch down, entering the step of removing the first batch down from the queue waiting for brushing down.
Preferably, after determining that the packet is the last packet, the method further includes:
judging whether the queue waiting for being refreshed is not an empty queue;
and if not, entering the step of obtaining a first lower brushing batch from the waiting lower brushing queue and judging whether the last packet in the first lower brushing batch is the last packet in the data block.
In order to solve the above technical problem, the present application further provides a packet writing alignment system, including:
the acquisition module is used for acquiring a packet sent by a client;
the first judgment module is used for judging whether the current lower brushing batch exceeds a lower brushing threshold value after the packet is added into the current lower brushing batch;
the first segmentation module is used for performing logic segmentation on the packet and marking the packet as segmented if the packet exceeds the threshold value, adding the part of the packet, which is smaller than the lower brushing threshold value, into the current lower brushing batch, and adding the rest part of the packet into the next lower brushing batch;
the second segmentation module is used for marking the packet as not segmented if the packet does not exceed the first segmentation module, and adding the packet into the current lower batch;
the second judgment module is used for judging whether the current brushing batch meets the brushing condition;
and the adding module is used for adding the current brushing batch into the queue waiting for brushing.
In order to solve the above technical problem, the present application further provides a packet writing and aligning device, including a memory for storing a computer program;
and the processor is used for implementing the steps of the packet writing and aligning method when the computer program is executed.
In order to solve the above technical problem, the present application further provides a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the steps of the packet write alignment method are implemented as described above.
The method for writing and aligning the packet packets includes the steps that the packet packets sent by a client are obtained, whether the current lower batch exceeds a lower brushing threshold value or not is judged after the packet packets are added into the current lower batch, if yes, the packet packets are logically segmented and marked as segmented, the part of the packet smaller than the lower brushing threshold value is added into the current lower batch, and the rest part of the packet is added into the next lower batch; if not, marking the packet as not segmented, adding the packet into the current lower batch, judging whether the current lower batch added into the waiting lower batch meets the lower brushing condition or not, and if so, brushing the current lower batch. Therefore, the writing alignment processing of the packet is realized, the packet is enabled to accord with the size of the lower brush through the segmentation processing of the packet, the part which is not in accordance with the size of the lower brush is processed in batches, the speed of writing in a magnetic disk can be improved, the possibility of messy codes is reduced, the time of reading data is reduced, and the occurrence of errors is reduced.
Drawings
In order to more clearly illustrate the embodiments of the present application, the drawings required for the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained by those skilled in the art without inventive effort.
Fig. 1 is a flowchart of a packet write alignment method provided in an embodiment of the present application;
fig. 2 is a flowchart for determining whether a current brushing batch meets the brushing condition according to the embodiment of the present application;
FIG. 3 is a flowchart of a process of waiting for a queue to be flushed according to an embodiment of the present disclosure;
fig. 4 is a flowchart of packet processing, and flushing and acknowledgement provided in the embodiment of the present application;
fig. 5 is a structural diagram of a packet write alignment system according to an embodiment of the present application;
fig. 6 is a structural diagram of a packet writing and aligning device according to another embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only some embodiments of the present application, and not all embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without any creative effort belong to the protection scope of the present application.
The application aims to provide a packet writing alignment method, which can solve the problem that when data are written through an HDFS (Hadoop distributed File System), a packet is not aligned, brushed down and written into a disk.
In the packet writing and aligning method provided by the present application, the segmentation, the alignment operation, and the flushing action of the packet are implemented by a controller of the server, for example, the controller may be a Micro Controller Unit (MCU), and may of course be implemented by other controllers besides the MCU, which is not limited in the present application.
In order that those skilled in the art will better understand the disclosure, the following detailed description will be given with reference to the accompanying drawings.
Fig. 1 is a flowchart of a packet write alignment method provided in an embodiment of the present application, and as shown in fig. 1, the method includes:
s10: and acquiring a packet sent by the client.
Specifically, in this embodiment, the client sends a Block to the data node, where the Block is split into a plurality of packet data packets and sent to the server, where a sequence number (seqno), a write position (offset), and a write data length (length) are recorded in the packet, and the packet default size is 64k. For example, a Block with a size of 1M may be split into 16 packets with a size of 64k for transmission to the server. And the server can start to store after acquiring the packet with the upgrading information. It should be noted that the above mentioned packet size and Block size are for better understanding by those skilled in the art, and do not represent only this embodiment, and the present embodiment does not specifically limit the packet size and the Block size.
Therefore, the data can be processed in batches by segmenting the larger Block into a smaller packet, and the method is more convenient and faster. The failure of one whole Block transmission is prevented from causing the failure of the whole flow.
S11: and judging whether the current lower brushing batch exceeds a lower brushing threshold value after the packet is added into the current lower brushing batch. If yes, entering the step S12; if not, the process proceeds to step S13.
Specifically, in this embodiment, the MCU determines whether the current packet added to the current lower brushing batch will exceed the lower brushing threshold. The lower brushing threshold is preset in advance for the current lower brushing batch, when the lower brushing threshold is exceeded, the current batch of data is full, the next batch of received packet packets need to be replaced, the packets are distributed according to the lower brushing threshold, the packet packets can be aligned, the speed of writing in a disk can be increased, and the possibility of messy codes is reduced. For example, the preset lower brushing threshold is 4M, the size of the obtained packet is 5M at this time, that is, the lower brushing threshold is exceeded, the previous 4M data of the packet needs to be stored into the current lower brushing batch, the current lower brushing batch is full at this time, the remaining 1M is stored into the next lower brushing batch, the next lower brushing batch contains the last 1M in the packet at this time, and the size of the next obtained packet is compared in the remaining 3M storage space of the next lower brushing batch. If the obtained packet is 1M and the current lower batch has a storage space larger than 1M, the current packet can be directly stored in the current lower batch. It should be noted that, the lower-brushing threshold and the packet size are both intended to make the skilled person better understand, and do not represent only this embodiment, and the present embodiment does not make any special limitation on the packet size and the lower-brushing threshold.
Therefore, the alignment size is limited by setting the lower brushing threshold, so that the lower brushing threshold of the received packet is brushed in batches, thereby reducing messy codes and improving the writing speed of the disk.
S12: and logically segmenting the packet and marking the packet as segmented, adding the part of the packet smaller than the lower brushing threshold value into the current lower brushing batch, and adding the rest part into the next lower brushing batch.
Specifically, in this embodiment, if a packet is larger than the remaining storage size in the current lower batch, the packet needs to be logically segmented. It should be noted that the logical segmentation is not equal to the traditional segmentation, and because the particularity of a packet is to ensure that data is not lost, a portion of the packet that exceeds the lower brushing threshold needs to be marked as a segmented portion, but the packet is still a whole at this time, and packet split data is not lost due to the logical segmentation. For example, when the lower scrub threshold is 4M, at this time, there is 1M packet1 in the current lower scrub, the remaining storage space is 3M, the mcu obtains 5M packet2, at this time, the boundary between the first 3M and the second 2M in the packet is used as a boundary point between the current lower scrub and the next lower scrub, at this time, the current lower scrub includes the first 3M of packet1 and packet2, and the next lower scrub includes the second 2M of packet2, it needs to be noted that, although two parts of packet2 are in different lower scrubs, due to the particularity of the packet and the continuity of the lower scrub queue, they are not separated in a physical sense, but are logically separated according to the lower scrub threshold, so as to prepare for aligning the lower scrub writing to the disk. It should be noted that, the lower-brushing threshold and the packet size are both intended to make the skilled person better understand, and do not represent only this embodiment, and the present embodiment does not make any special limitation on the packet size and the lower-brushing threshold.
Therefore, the obtained packet is logically segmented according to the lower brushing threshold value, so that the integrity of the packet is guaranteed, the lower brushing can be aligned when the packet is brushed, and the speed of writing in a disk is increased while data is protected.
S13: and marking the packet as not segmented, and adding the packet to the current lower batch.
Specifically, in this embodiment, if a packet is smaller than the remaining storage size in the current lower batch, the packet does not need to be logically split, and the obtained packet may be directly marked as being unsingulated and added to the current lower batch. For example, the lower brushing threshold is 4M, and the current lower brushing batch is empty at this time, that is, there is 4M storage space, the obtained packet is 3M, and the packet is smaller than the storage space of the current lower brushing batch at this time, and may be directly added to the current lower brushing batch to wait for the lower brushing. It should be noted that the lower brushing threshold and the packet size are both for better understanding by those skilled in the art, and do not represent only this embodiment, and the embodiment does not make specific limitations on the packet size and the lower brushing threshold.
Therefore, by comparing the size of the packet with the remaining storage space of the current lower batch, the packet is smaller than the storage space of the current lower batch, and is directly added to the current lower batch to wait for the lower batch to be brushed, so that the speed of writing in the disk is improved.
S14: and judging whether the current brushing batch meets the brushing condition or not. If yes, the process proceeds to step S15.
Specifically, in this embodiment, by determining whether the current lower batch meets a lower brushing condition, the lower brushing condition includes that the current lower batch is full, the obtained packet is the first in Block, the obtained packet is the last in Block, and the obtained packet is smaller than the largest packet size in the queue, and if the current lower batch meets the lower brushing condition, the current batch may be added to the queue waiting for lower brushing.
S15: and adding the current brushing batch into a queue waiting for brushing.
Specifically, through brushing down and criticizing and screening in this embodiment, brush down to brushing down and criticizing the brush down that accords with down the brush condition, can make down the brush accord with predetermined requirement, reduced the possibility that the messy sign indicating number appears, increased the speed of brushing down.
It can be seen that the packet writing alignment method provided by the present application specifically includes obtaining a packet sent by a client, determining whether a current lower batch exceeds a lower batch threshold value after adding the packet to the current lower batch, if so, performing logical segmentation on the packet and marking the packet as segmented, adding a portion of the packet smaller than the lower batch threshold value to the current lower batch, and adding the remaining portion to the next lower batch; if not, marking the packet as not segmented, adding the packet into the current lower brushing batch, judging whether the current lower brushing batch meets the brushing-down condition, and if so, adding the current lower brushing batch into a waiting brushing-down queue. Therefore, the writing alignment processing of the packet is realized, the packet accords with the size of the lower brush through the segmentation processing of the packet, the parts which do not accord with the size of the lower brush are processed in batches, the speed of writing in a magnetic disk can be improved, the possibility of messy codes is reduced, the time of reading data is reduced, and the occurrence of errors is reduced.
On the basis of the above embodiment, as a preferred embodiment, the performing logic splitting on a packet includes:
and marking the part of the packet exceeding the lower brushing threshold as the sliced part.
Specifically, in this embodiment, if a packet is larger than the remaining storage size in the current lower batch, the packet needs to be logically segmented. It should be noted that the logical segmentation is not equal to the traditional segmentation, and because the particularity of a packet is to ensure that data is not lost, a portion of the packet that exceeds the lower brushing threshold needs to be marked as a segmented portion, but the packet is still a whole at this time, and packet split data is not lost due to the logical segmentation.
Therefore, the obtained packet is logically segmented according to the lower brushing threshold, so that the integrity of the packet is ensured, the lower brushing can be aligned during the lower brushing, and the speed of writing in a disk is increased while data is protected.
On the basis of the foregoing embodiment, as a preferred embodiment, fig. 2 is a flowchart for determining whether a current brushing batch meets the brushing-down condition provided in the embodiment of the present application, and as shown in fig. 2, the condition includes:
s16: and obtaining the maximum packet size in the current lower brushing batch.
Specifically, in this embodiment, after acquiring a new packet, the MCU needs to update the largest packet size of all packets in the current lower batch, for example, the current lower batch includes a packet1 of size 2m and a packet2 of size 3M, and the largest packet is packet2 at this time. It should be noted that, in this embodiment, the size and the number of packet packets in the current lower brushing batch are not particularly limited.
S17: and judging whether the sequence number of the packet and the offset in the data block are both 0 or not. If both are 0, the process proceeds to step S18.
Specifically, in this embodiment, it is determined whether the current packet is the first packet in the first next round in the Block by determining whether the sequence number of the packet and the offset in the data Block are both 0, for example, if the packet is the first packet in the first next round, the sequence number and the offset in the Block are both 0, if the packet is the second packet in the first next round, the previous packet is 64k, the sequence number of the packet is 1, and the offset is 64k. It should be noted that, the sequence number and the offset are only used for better understanding by those skilled in the art, and do not represent only this embodiment, and the sequence number and the offset of the packet are not particularly limited in this example.
Therefore, if only the serial number or only the offset cannot be determined whether the current packet is the first data packet in the first next batch in the Block, whether the current packet is the first data packet in the first next batch in the Block is determined to be more accurate and specific by determining whether the conditions are met simultaneously.
S18: and adding the packet into a waiting lower brushing queue, and replacing a new lower brushing batch as a next lower brushing batch.
Specifically, a packet with a sequence number and an offset of 0 is added into a queue waiting for flushing to directly wait for flushing, a next packet is not waited for to be filled with a flushing threshold, a new lower flushing batch is taken as a next lower flushing batch again, and a new packet is waited for.
Therefore, after the packet is judged to be the first packet, as the rest packets in the queue do not interfere with the first packet, the flushing can be directly carried out, and the waiting time is saved.
S19: and judging whether the packet is segmented or not. If the segmentation is carried out, the step S20 is carried out; if not, the process proceeds to step S24.
Specifically, whether the current data packet is a logically segmented packet is judged, if the current data packet is segmented, the packet is separated from the segmentation point, a part smaller than a preset lower brushing threshold value is added into the current lower brushing batch, and the rest part, namely the segmented part, is added into the next lower brushing batch. And if the packet is not segmented, directly adding the packet into the current lower batch.
S20: and adding the part of the packet, which is smaller than the preset lower brushing threshold value, into the current lower brushing batch, and adding the segmented part into the next lower brushing batch.
If the packet is segmented, separating the packet from the segmentation point, adding the part smaller than the preset lower brushing threshold value into the current lower brushing batch, and adding the rest part, namely the segmented part, into the next lower brushing batch.
S21: and judging whether the data length of the packet is smaller than the maximum packet size in the lower batch where the packet is located. If the value is less than the preset value, the step S22 is executed; if not, the process proceeds to step S23.
S22: adding the packet into the next lower brushing batch;
s23: adding a packet into the current lower batch;
s24: and adding the packet into the current lower brushing batch.
In this embodiment, by determining whether the data length of a packet is smaller than the maximum packet size in the lower brushing batch where the packet is located, if the data length is smaller than the maximum packet size, it can be considered that the packet of the current batch is completely transmitted because the packet represents that the packet is not the same as the packet in the current batch, and the packet can be added to the next lower brushing batch; otherwise, the packet is added to the current lower brushing batch.
Therefore, the lower brush batch which meets the lower brush condition is screened and brushed, so that the lower brush can meet the preset requirement, the possibility of messy codes is reduced, and the brushing speed is increased.
On the basis of the foregoing embodiment, as a preferred embodiment, as shown in fig. 2, after determining that the sequence number of the packet and the offset in the data block where the packet is located are not both 0, the method further includes:
s25: and judging whether the data length of the packet is smaller than the maximum packet size in the lower batch where the packet is located. If the value is smaller than the predetermined value, the process proceeds to step S18.
Specifically, in this embodiment, by determining whether the data length of the current packet is smaller than the largest packet size of all packets in the current lower brushing batch where the current packet is located, for example, the size of packet1 included in the current lower brushing batch is 2m, the size of packet2 is 3M, the largest packet is packet2 at this time, the size of received packet3 is 2M at this time, that is, packet3 is directly added to the queue waiting for lower brushing, and a new lower brushing batch is replaced as the next lower brushing batch. In addition, if the size of packet3 is larger than the storage space of the current lower batch, the logical segmentation is also needed, and the rest part is added into the next lower batch. It should be noted that, in this embodiment, the size and the number of packet packets in the current lower brushing batch are not particularly limited.
It can be seen that, by determining whether the data length of a packet is smaller than the size of the largest packet in the lower batch where the packet is located, if so, it represents that the current packet changes, and is different from the largest packet, that is, a packet smaller than the largest packet can be directly added to the lower batch, otherwise, the current lower batch is added.
On the basis of the foregoing embodiment, as a preferred embodiment, as shown in fig. 2, after determining that the data length of a packet is not smaller than the maximum packet size in the lower batch where the packet is located, the method further includes:
s26: and judging whether the packet is segmented or not. If the dicing has been performed, the process proceeds to step S27.
S27: adding the part of the packet smaller than the preset lower brushing threshold value into the current lower brushing batch, and adding the segmented part into the next lower brushing batch;
s28: judging whether the packet is segmented or not, if not, entering the step S29; if the dicing has been performed, the process proceeds to step S30.
S29: adding a packet into the current lower batch;
s30: the cut portion is added to the next lower batch.
Specifically, in this embodiment, by determining whether the current packet is segmented, if so, adding the part of the packet smaller than the preset lower brushing threshold to the current lower brushing batch and adding the segmented part to the next lower brushing batch according to the logical segmentation result and the segmentation point. And if the packet is not segmented, directly adding the packet into the current lower brushing batch.
Therefore, the obtained packet is logically segmented according to the lower brushing threshold value, so that the integrity of the packet is guaranteed, the lower brushing can be aligned when the packet is brushed, and the speed of writing in a disk is increased while data is protected.
On the basis of the foregoing embodiment, as a preferred embodiment, fig. 2 is a flowchart after determining that a packet is not fragmented, which is provided in the embodiment of the present application, and as shown in fig. 2, after determining that a packet is not fragmented, the method further includes:
s31: and judging whether the current lower brushing batch of the packet exceeds a lower brushing threshold value or not. If yes, the process proceeds to step S32.
S32: and adding the part of the current lower brushing batch exceeding the lower brushing threshold value into the waiting lower brushing queue, adding the rest part into the next lower brushing batch, and entering the step S28.
Specifically, if the current lower brushing batch where the packet is located exceeds the lower brushing threshold, the part exceeding the lower brushing threshold still needs to be logically segmented and processed in batches, and whether the current lower brushing batch exceeds the lower brushing threshold can be determined through the following inequality: batchOffset-alginOffset + batchLen is not less than FLUSH _ SIZE, wherein batchOffset is the offset of data in the current batch in a file, alginOffset is the offset of the current batch after alignment, batchLen is the data length in the current batch, and FLUSH _ SIZE is the lower brushing SIZE. And judging whether the current lower batch where the packet is still over the lower brushing threshold value after the first segmentation is carried out through the inequality, if so, carrying out secondary logic segmentation on the current lower batch where the packet is, adding the part which is smaller than the lower brushing threshold value in the segmented part into the current lower batch after the segmentation, and adding the rest part into the next lower batch.
Therefore, the current lower batch is judged for the second time, so that the condition that the lower brushing processing cannot be carried out due to the fact that the current lower batch is still larger than the lower brushing threshold value after the first time of segmentation is avoided, and the current lower batch still exceeding the lower brushing threshold value is subjected to the second time logical segmentation processing.
On the basis of the above embodiment, as a preferred embodiment, as shown in fig. 2, after judging that the segmented portion is smaller than the lower brushing threshold, the method further includes:
s33: and judging whether the packet is the last packet in the data block. If the packet is the last packet, the step S34 is carried out; if not, the process proceeds to step S35.
S34: and adding the packet into a waiting brushing queue, and replacing a new brushing batch as a next brushing batch.
S35: and adding the packet into the current lower brushing batch.
Specifically, whether a packet is the last packet in a Block is judged, if the packet is the last packet, the packet is directly added into a queue waiting for lower batch flushing, a new lower batch is replaced to be used as the next lower batch flushing, and otherwise, the packet is added into the current lower batch flushing but the new lower batch flushing is not replaced.
Therefore, whether the next packet needs to be waited continuously is judged by judging whether the packet is the last packet in the data block, so that useless waiting time is reduced, and the brushing efficiency is improved.
On the basis of the foregoing embodiment, as a preferred embodiment, fig. 3 is a flowchart of a queue waiting for being flushed processing provided in the embodiment of the present application, and as shown in fig. 3, the method includes:
s36: and judging whether the packet is not the last packet. If not, the process proceeds to step S37.
S37: and acquiring a first lower brushing batch from the waiting lower brushing queue, and judging whether the last packet in the first lower brushing batch is the last packet in the data block. If the last one is obtained, the process proceeds to step S38.
S38: and judging whether the first lower batch only has one packet or not. If only one, the process proceeds to step S39; if there is more than one, the process proceeds to step S40.
S39: and adding the first batch to the confirmation collection to wait for the confirmation of the client, returning the confirmation to the client, removing the first batch from the queue waiting for the batch, and returning to the step S36.
S40: and adding the first batch to the confirmation collection to wait for the confirmation of the client, after the first batch is brushed completely, removing the first batch from the waiting brushing queue, and returning to the step S36.
If not, the process proceeds to step S40.
Specifically, in this embodiment, when the last packet in the current lower Batch (Batch) is the last packet in Block and there is only one packet in Batch, that is, the client receives the acknowledgement information of all data packet, and indicates that the data has been written, and the packet may directly return an acknowledgement. And taking out the first Batch from the current Batch queue when judging that the packet is not the last packet, judging whether the last packet in the Batch is the last packet in the Block, if so, continuously judging whether the first Batch is only one packet, if so, adding the first next Batch into the confirmation collection to wait for confirmation of the client, returning confirmation to the client, removing the first next Batch from the waiting next Batch queue, and returning to the step of S36. Otherwise, adding the first lower batch into the confirmation collection to wait for the confirmation of the client, after finishing the lower batch brushing of the first lower batch, entering the step of removing the first lower batch from the waiting lower batch queue, and similarly returning to the step of S36. If not the last, the process proceeds to step S37.
It can be seen that, by confirming that the last packet in the current Batch is the last packet in the Block, and when only one packet is in the Batch, that is, the client receives the confirmation information of all data packet packets, it indicates that the data has been written, and the packet may directly return a confirmation.
On the basis of the foregoing embodiment, as a preferred embodiment, as shown in fig. 3, after determining that a packet is the last packet, the method further includes:
and judging whether the queue waiting for being refreshed is not an empty queue.
If not, the process proceeds to step S37.
After the packet is judged to be the last packet, whether the current waiting flushing is an empty queue needs to be judged, if not, the current waiting flushing is represented by that a packet needs to be flushed, and the step returns to the step S37 for judgment. And if the packet is empty, the lower brushing is finished, and no new packet needs to be brushed down. It should be noted that, the lower brush adopted in this embodiment is an asynchronous lower brush, the confirmation is realized through a callback after the lower brush is completed, and when the callback completed by the lower brush is received, the Batch is taken out from the confirmation set through the Batch offset of the Batch that is sent in by the callback, and all packets in the Batch are confirmed. There is a special case here, because we reorganize a packet in the above process, a part of the packet is split, and two batchs are added, so that confirmation can be performed after callbacks of the two batchs are received, the number of times (shouldAck) that the packet should be confirmed records the number of times that the packet needs to receive a callback, and the packet is ack to the client only when the number is 0.
Fig. 4 is a flowchart of Packet processing, flushing and acknowledgement provided in this embodiment, as shown in fig. 4, where a client (1) sends a Packet to a data node (DataNode), the DataNode includes a data block receiving end (BlockReceiver) and a Packet receiving end (PacketReceiver), the PacketReceiver (2) receives and parses the Packet, (3) reorganizes the Packet and adds the Packet to a current lower batch, where (3) corresponds to the content steps in fig. 1 and fig. 2, (4) adds a lower flush queue, corresponds to the steps in fig. 3, and after the lower flush queue is sorted, (5) aligns the lower flush data (6) and adds the lower flush data to a queue to be acknowledged (7), and then calls back, and a reply end (Responder) is used in the acknowledgement queue (8) to reply an Ack (acknowledgement) to the client (9) and receive a Packet write-in Ack, which indicates that the Packet is written in an Ack.
In the above embodiments, the packet write alignment method is described in detail, and the present application also provides an embodiment corresponding to the packet write alignment system. It should be noted that the present application describes embodiments of the system part from two perspectives, one from the perspective of the function module and the other from the perspective of the hardware.
Based on the angle of the functional module, this embodiment provides a packet writing and aligning system, and fig. 5 is a structural diagram of the packet writing and aligning system provided in this embodiment, as shown in fig. 5, the system includes:
an obtaining module 10, configured to obtain a packet sent by a client;
the first judging module 11 is configured to judge whether the current lower brushing batch exceeds a lower brushing threshold after the packet is added to the current lower brushing batch;
the first segmentation module 12 is configured to, if the number of the packet packets exceeds the threshold value, perform logical segmentation on the packet packets and mark the packet packets as segmented, add a portion of the packet packets smaller than the lower brushing threshold value to the current lower brushing batch, and add the remaining portion to the next lower brushing batch;
the second segmentation module 13 is configured to mark the packet as not segmented if the packet does not exceed the first segmentation module, and add the packet to the current lower batch;
the second judging module 14 is configured to judge whether the current brushing batch meets the brushing condition;
and the adding module 15 is used for adding the current brushing batch into the queue waiting for brushing.
Since the embodiment of the system part corresponds to the embodiment of the method part, the embodiment of the system part is described with reference to the embodiment of the method part, and is not repeated here.
The packet writing and aligning system provided by the embodiment corresponds to the method, and therefore has the same beneficial effects as the method.
Based on the hardware perspective, fig. 6 is a structural diagram of a packet writing alignment apparatus according to another embodiment of the present application, as shown in fig. 6, including a memory 20 for storing a computer program;
a processor 21, configured to implement the steps of the packet write alignment method as described above when executing the computer program.
The packet writing alignment apparatus provided in this embodiment may include, but is not limited to, a server, and the like.
The processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The Processor 21 may be implemented in at least one hardware form of a Digital Signal Processor (DSP), a Field-Programmable Gate Array (FPGA), and a Programmable Logic Array (PLA). The processor 21 may also include a main processor and a coprocessor, where the main processor is a processor for Processing data in an awake state, and is also called a Central Processing Unit (CPU); a coprocessor is a low power processor for processing data in a standby state. In some embodiments, the processor 21 may be integrated with a Graphics Processing Unit (GPU) which is responsible for rendering and drawing the content required to be displayed by the display screen. In some embodiments, the processor 21 may further include an Artificial Intelligence (AI) processor for processing computing operations related to machine learning.
The memory 20 may include one or more computer-readable storage media, which may be non-transitory. Memory 20 may also include high speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In this embodiment, the memory 20 is at least used for storing the following computer program 201, wherein after being loaded and executed by the processor 21, the computer program can implement the relevant steps of the packet writing alignment method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202, data 203, and the like, and the storage manner may be a transient storage manner or a permanent storage manner. Operating system 202 may include, among others, windows, unix, linux, and the like. Data 203 may include, but is not limited to, data related to packet write alignment methods, and the like.
In some embodiments, the packet writing alignment apparatus may further include a display 22, an input/output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.
Those skilled in the art will appreciate that the configuration shown in figure 6 does not constitute a limitation of the packet write alignment apparatus and may include more or fewer components than those shown.
The packet writing and aligning device provided by the embodiment of the application comprises a memory and a processor, and the processor can realize the method when executing the program stored in the memory.
Finally, the application also provides a corresponding embodiment of the computer readable storage medium. The computer-readable storage medium has stored thereon a computer program which, when being executed by a processor, carries out the steps as set forth in the above-mentioned method embodiments.
It is to be understood that if the method in the above embodiments is implemented in the form of software functional units and sold or used as a stand-alone product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium and executes all or part of the steps of the methods described in the embodiments of the present application, or all or part of the technical solutions. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The above provides a detailed description of a packet writing alignment method, system, device and medium. The embodiments are described in a progressive manner in the specification, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed in the embodiment corresponds to the method disclosed in the embodiment, so that the description is simple, and the relevant points can be referred to the description of the method part. It should be noted that, for those skilled in the art, it is possible to make several improvements and modifications to the present application without departing from the principle of the present application, and such improvements and modifications also fall within the scope of the claims of the present application.
It should also be noted that, in this specification, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising a," "8230," "8230," or "comprising" does not exclude the presence of additional like elements in a process, method, article, or apparatus that comprises the element.
Claims (12)
1. A packet writing alignment method is characterized by comprising the following steps:
acquiring a packet sent by a client;
judging whether the current lower brushing batch exceeds a lower brushing threshold value after the packet is added into the current lower brushing batch;
if the current lower batch is not the lower batch, performing logic segmentation on the packet and marking the packet as segmented, adding the part of the packet, which is smaller than the lower brushing threshold value, into the current lower batch, and adding the rest part into the next lower batch;
if not, marking the packet as not segmented, and adding the packet into the current lower batch;
judging whether the current lower brushing batch meets a lower brushing condition;
and if so, adding the current batch refreshing into a queue waiting for refreshing.
2. The packet write alignment method according to claim 1, wherein the performing the logical segmentation on the packet includes:
marking the part of the packet exceeding the lower brushing threshold as a sliced part.
3. The packet write alignment method according to claim 2, wherein the determining whether the current lower brushing batch meets a lower brushing condition comprises:
acquiring the maximum packet size in the current lower brushing batch;
judging whether the sequence number of the packet and the offset in the data block are both 0 or not;
if the number of the packets is 0, adding the packet packets into the queue waiting for lower brushing, and replacing the new lower brushing batch as the next lower brushing batch;
judging whether the packet is segmented or not;
if the packet is segmented, adding the part of the packet, which is smaller than the preset lower brushing threshold value, into the current lower brushing batch, adding the segmented part into the next lower brushing batch, and judging whether the data length of the packet is smaller than the maximum packet size in the lower brushing batch where the packet is located;
if the packet size is smaller than the preset size, adding the packet into the next lower brushing batch;
if not, adding the packet into the current lower batch;
and if the packet is not segmented, adding the packet into the current lower batch.
4. The packet write alignment method according to claim 3, wherein after the determining that the sequence number of the packet and the offset in the data block where the packet is located are not both 0, the method further comprises:
judging whether the data length of the packet is smaller than the maximum packet size in the lower batch where the packet is located;
and if the number of the packets is smaller than the preset threshold, entering the step of adding the packet packets into the queue waiting for brushing down, and replacing the new brushing down batch as the next brushing down batch.
5. The packet write alignment method according to claim 4, wherein after the determining that the data length of the packet is not smaller than the maximum packet size in the lower batch where the packet is located, the method further comprises:
judging whether the packet is segmented or not;
if the packet is segmented, adding the part of the packet smaller than the preset lower brushing threshold value into the queue waiting for lower brushing, and adding the segmented part into the next lower brushing batch;
judging whether the packet is segmented or not;
if not, adding the packet into the current lower batch;
if so, adding the already cut portion to the next lower batch.
6. The packet write alignment method according to claim 5, further comprising, after the determining that the packet is not split:
judging whether the current lower brushing batch where the packet is located exceeds the lower brushing threshold value;
and if the current lower batch exceeds the lower brushing threshold, adding the part of the current lower batch exceeding the lower brushing threshold into the queue waiting for lower brushing, adding the rest part into the next lower batch, and entering the step of judging whether the packet is segmented or not.
7. The packet writing alignment method according to claim 6, wherein after determining that the current lower brushing batch where the packet is located is smaller than the lower brushing threshold, the method further comprises:
judging whether the packet is the last packet in the data block;
if the packet is the last packet, adding the packet into the queue waiting for lower batch flushing, replacing the new lower batch as the next lower batch flushing, and entering the step of judging whether the packet is segmented or not;
and if the packet is not the last packet, adding the packet into the current lower batch.
8. The packet write alignment method according to claim 1, wherein after adding the current lower flushing batch into a waiting lower flushing queue, further comprising:
judging whether the packet is not the last packet;
if not, acquiring a first lower brushing batch from the waiting lower brushing queue, and judging whether the last packet in the first lower brushing batch is the last packet in the data block;
if the packet is the last packet, judging whether the first lower batch only has one packet;
if only one packet is found, adding the first lower batch to a confirmation collection to wait for the confirmation of the client, returning confirmation to the client, removing the first lower batch from the queue waiting for lower batch, and returning to the step of judging whether the packet is not the last packet;
if the number of the lower batch packets is more than one, adding the first lower batch packet into a confirmation collection to wait for the confirmation of the client, entering the step of removing the first lower batch packet from the waiting lower batch packet after the lower batch packet is finished, and returning to the step of judging whether the packet is not the last packet;
and if the number of the next batch is not the last batch, adding the first batch to a confirmation collection to wait for the confirmation of the client, and after the first batch is refreshed, entering the step of removing the first batch from the waiting batch refreshing queue.
9. The packet write alignment method according to claim 8, wherein after determining that the packet is the last packet, the method further comprises:
judging whether the queue waiting for being refreshed is not an empty queue;
and if not, entering the step of obtaining a first lower brushing batch from the waiting lower brushing queue and judging whether the last packet in the first lower brushing batch is the last packet in the data block.
10. A packet write alignment system, comprising:
the acquisition module is used for acquiring a packet sent by a client;
the first judgment module is used for judging whether the current lower brushing batch exceeds a lower brushing threshold value after the packet is added into the current lower brushing batch;
the first segmentation module is used for performing logic segmentation on the packet and marking the packet as segmented if the packet exceeds the threshold value, adding the part of the packet, which is smaller than the lower brushing threshold value, into the current lower brushing batch, and adding the rest part of the packet into the next lower brushing batch;
the second segmentation module is used for marking the packet as not segmented if the packet does not exceed the current lower batch, and adding the packet into the current lower batch;
the second judging module is used for judging whether the current brushing batch meets the brushing condition or not;
and the adding module is used for adding the current brushing batch into a queue waiting for brushing.
11. A packet writing and aligning device is characterized by comprising a memory, a data processing unit and a data processing unit, wherein the memory is used for storing a computer program;
a processor for implementing the steps of the packet write alignment method as claimed in any one of claims 1 to 9 when executing the computer program.
12. A computer-readable storage medium, having stored thereon a computer program which, when executed by a processor, performs the steps of the packet write alignment method of any of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211348879.XA CN115629711A (en) | 2022-10-31 | 2022-10-31 | Packet writing alignment method, system, device and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211348879.XA CN115629711A (en) | 2022-10-31 | 2022-10-31 | Packet writing alignment method, system, device and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115629711A true CN115629711A (en) | 2023-01-20 |
Family
ID=84908544
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211348879.XA Pending CN115629711A (en) | 2022-10-31 | 2022-10-31 | Packet writing alignment method, system, device and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115629711A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117389484A (en) * | 2023-12-12 | 2024-01-12 | 深圳大普微电子股份有限公司 | Data storage processing method, device, equipment and storage medium |
-
2022
- 2022-10-31 CN CN202211348879.XA patent/CN115629711A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117389484A (en) * | 2023-12-12 | 2024-01-12 | 深圳大普微电子股份有限公司 | Data storage processing method, device, equipment and storage medium |
| CN117389484B (en) * | 2023-12-12 | 2024-04-26 | 深圳大普微电子股份有限公司 | Data storage processing method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115629711A (en) | Packet writing alignment method, system, device and medium | |
| CN113703951A (en) | Method and device for processing DMA (direct memory Access) and computer readable storage medium | |
| CN114995759A (en) | Barrel fragment processing method, device, equipment and medium | |
| CN112650449B (en) | Method and system for releasing cache space, electronic device and storage medium | |
| CN112363980A (en) | Data processing method and device for distributed system | |
| CN115934999A (en) | Video stream data storage method, device and medium based on block file | |
| CN115437572A (en) | Data dropping method, device, equipment and medium | |
| CN115480973A (en) | Server performance test method, device and medium | |
| CN111309475B (en) | Detection task execution method and equipment | |
| CN114721876A (en) | Data backup method, device and medium | |
| CN114788294A (en) | Content pushing method, device, server and storage medium | |
| CN113703678A (en) | Method, device, equipment and medium for re-splitting index of storage bucket | |
| CN112883239A (en) | Resource allocation method and device, computer equipment and storage medium | |
| CN111090633A (en) | Small file aggregation method, device and equipment of distributed file system | |
| CN114301927B (en) | Main node selection method, device and medium in distributed system | |
| CN110365342A (en) | Waveform decoder method and device | |
| CN113126918B (en) | Fragmented data reading method, device and medium | |
| CN113076178B (en) | Message storage method, device and equipment | |
| CN114466022B (en) | Method, device and medium for obtaining server seed nodes in cluster | |
| CN112104907B (en) | Picture playing method and device | |
| CN117727353A (en) | Storage block screening method, device, equipment and medium based on solid state disk | |
| CN114676008A (en) | Omap operation performance testing method, apparatus and medium | |
| CN114968208A (en) | Object recovery method, device and medium | |
| CN115408130A (en) | Task execution method and device, storage medium and electronic device | |
| CN114443767A (en) | Method, apparatus, device and medium for determining consistency level of distributed system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |