CN111563017B - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN111563017B CN111563017B CN202010348381.8A CN202010348381A CN111563017B CN 111563017 B CN111563017 B CN 111563017B CN 202010348381 A CN202010348381 A CN 202010348381A CN 111563017 B CN111563017 B CN 111563017B
- Authority
- CN
- China
- Prior art keywords
- log
- stream
- identifier
- block
- log stream
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 25
- 238000000034 method Methods 0.000 claims abstract description 23
- 238000012545 processing Methods 0.000 claims description 14
- 238000013507 mapping Methods 0.000 claims description 6
- 230000002776 aggregation Effects 0.000 claims description 5
- 238000004220 aggregation Methods 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 4
- 230000004931 aggregating effect Effects 0.000 claims description 3
- 230000008569 process Effects 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 6
- 239000000872 buffer Substances 0.000 description 5
- 238000012217 deletion Methods 0.000 description 5
- 230000037430 deletion Effects 0.000 description 5
- 239000012634 fragment Substances 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000010076 replication Effects 0.000 description 1
- 238000010561 standard procedure Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The disclosure provides a data processing method and device. In the method, a logic storage space is set for all log streams in one log file, after log records of a certain log stream are acquired, the acquired log records are stored in the logic storage space corresponding to the log stream, namely, the fact that the log records corresponding to all log streams managed by a log storage system are stored in the same log file of a disk is achieved.
Description
Technical Field
The disclosure relates to the field of log storage processing, and in particular relates to a data processing method and device.
Background
At present, a distributed storage system is vigorously developed, in order to improve the stability and reliability of the system, a plurality of data backups, that is, a plurality of copies are arranged in the distributed storage system, and therefore, it is inevitable in the distributed storage system to ensure that the data stored by each copy is consistent. At present, the consistency of the data in each copy is often ensured through a consistency algorithm. For example, using the common consistency protocol, raft, a Leader node is elected in the distributed storage system and is responsible for managing log replication to achieve multi-copy consistency.
When implementing the above data consistency algorithm, a Write-ahead logging (WAL) system is relied on, which is a standard method for guaranteeing data integrity. In order to ensure the integrity of system data in a distributed storage system, operations on the system are often written into the WAL, so that the throughput performance of the WAL directly determines the writing performance of the whole distributed storage system, and therefore, how to improve the throughput of the WAL is a technical problem to be solved at present.
Disclosure of Invention
Accordingly, the present disclosure provides at least a data processing method and apparatus.
In a first aspect, the present disclosure provides a data processing method, including:
respectively acquiring a plurality of log records corresponding to each log stream in a log storage system;
determining a logic storage space corresponding to each log stream; wherein, the logic storage space corresponding to all log streams belongs to one log file;
and respectively storing the plurality of log records corresponding to each log stream into a corresponding logic storage space.
In one possible implementation manner, the determining the logical storage space corresponding to each log stream includes:
setting a first identifier for identifying a logic storage space of each log stream for the log stream respectively;
respectively establishing a mapping relation between a first identifier corresponding to each log stream and a logic storage space of the log stream;
the storing the plurality of log records corresponding to each log stream into the corresponding logic storage space respectively includes:
acquiring a first identifier corresponding to each log stream;
for each log stream, identifying each log record corresponding to the log stream with a first identifier of the log stream.
In one possible implementation manner, the storing the plurality of log records corresponding to each log stream in a corresponding logic storage space respectively further includes:
for each log stream, aggregating the plurality of log records corresponding to the log stream to obtain a log block corresponding to the log stream;
and storing the log blocks corresponding to the log streams into the log file.
In one possible implementation manner, the storing the plurality of log records corresponding to each log stream in a corresponding logic storage space respectively further includes:
carrying out aggregation processing on the log blocks corresponding to each log stream to obtain a log packet;
and storing the log package into the log file.
In one possible implementation manner, the data processing method further includes:
determining a second identifier for each log block and identifying each log block with the second identifier;
for each log record, determining a third identifier of the log record based on an order of generation of the log record in a log stream to which the log record belongs;
and identifying the corresponding log record by using the third identifier.
In one possible implementation manner, the data processing method further includes:
receiving a new log block to be written;
determining a target log stream to which the new log block belongs;
determining a largest second identifier in a log block corresponding to the target log stream stored in the log storage system;
and if the second identifier of the new log block is smaller than the maximum second identifier, setting the log block, of which the second identifier is larger than or equal to the second identifier of the new log block, in the log blocks corresponding to the target log stream stored in the log storage system as an invalid state.
In one possible implementation manner, the data processing method further includes:
receiving a new log block to be written;
determining a target log stream to which the new log block belongs;
determining a largest second identifier in a log block corresponding to the target log stream stored in the log storage system;
and if the second identifier of the new log block is larger than the maximum second identifier, setting the log block corresponding to the target log stream stored in the log storage system as a failure state.
In a second aspect, the present disclosure provides a data processing apparatus comprising:
the log acquisition module is used for respectively acquiring a plurality of log records corresponding to each log stream in the log storage system;
the storage determining module is used for determining a logic storage space corresponding to each log stream; wherein, the logic storage space corresponding to all log streams belongs to one log file;
and the storage module is used for respectively storing the plurality of log records corresponding to each log stream into the corresponding logic storage space.
In one possible implementation manner, the storage determining module is configured to, when determining the logical storage space corresponding to each log stream:
setting a first identifier for identifying a logic storage space of each log stream for the log stream respectively;
respectively establishing a mapping relation between a first identifier corresponding to each log stream and a logic storage space of the log stream;
the storage module is used for storing the plurality of log records corresponding to each log stream into the corresponding logic storage space respectively, wherein the storage module is used for:
acquiring a first identifier corresponding to each log stream;
for each log stream, identifying each log record corresponding to the log stream with a first identifier of the log stream.
In a third aspect, the present disclosure provides an electronic device comprising: a processor, a memory and a bus, said memory storing machine readable instructions executable by said processor, said processor and said memory communicating over the bus when the electronic device is running, said machine readable instructions when executed by said processor performing the steps of the data processing method as described above.
In a fourth aspect, the present disclosure also provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of a data processing method as described above.
The apparatus, electronic device, and computer-readable storage medium of the present disclosure include at least technical features substantially the same as or similar to technical features of any one embodiment of any one of the above-described methods of the present disclosure.
The disclosure provides a data processing method and device, wherein a plurality of log records corresponding to each log stream in a log storage system are firstly obtained respectively; then, determining a logic storage space corresponding to each log stream; wherein, the logic storage space corresponding to all log streams belongs to one log file; and finally, storing the plurality of log records corresponding to each log stream into the corresponding logic storage space. In the method, a logic storage space is respectively set for all log streams in one log file, after log records of a certain log stream are acquired, the acquired log records are stored in the logic storage space corresponding to the log stream, namely, the fact that log records corresponding to all log streams managed by a log storage system are stored in the same log file of a disk is achieved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings that are needed in the embodiments will be briefly described below, it being understood that the following drawings only illustrate some embodiments of the present disclosure and therefore should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person of ordinary skill in the art.
FIG. 1 illustrates a flow chart of a data processing method provided by an embodiment of the present disclosure;
FIG. 2 illustrates a schematic diagram of a log storage system provided by an embodiment of the present disclosure;
FIG. 3 illustrates a flow chart of another data processing method provided by an embodiment of the present disclosure;
FIG. 4 illustrates a flow chart for writing log records into log files provided by embodiments of the present disclosure;
FIG. 5 illustrates a flow chart for specifically writing log records into log files provided by embodiments of the present disclosure;
FIG. 6 illustrates a flow chart for setting identifiers provided by embodiments of the present disclosure;
FIG. 7 is a schematic diagram of a data processing apparatus according to an embodiment of the present disclosure;
fig. 8 illustrates a schematic structural diagram of an electronic device according to an embodiment of the present disclosure;
fig. 9 is a schematic diagram showing the structure of a log storage system in the prior art.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions of the embodiments of the present disclosure will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present disclosure, it should be understood that the drawings in the present disclosure are for the purpose of illustration and description only, and are not intended to limit the scope of protection of the present disclosure. In addition, it should be understood that the schematic drawings are not drawn to scale. A flowchart, as used in this disclosure, illustrates operations implemented according to some embodiments of the present disclosure. It should be understood that the operations of the flow diagrams may be implemented out of order and that steps without logical context may be performed in reverse order or concurrently. Moreover, one or more other operations may be added to or removed from the flow diagrams by those skilled in the art in light of the present disclosure.
In addition, the described embodiments are only some, but not all, of the embodiments of the present disclosure. The components of the embodiments of the present disclosure, which are generally described and illustrated in the figures herein, may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present disclosure provided in the accompanying drawings is not intended to limit the scope of the disclosure, as claimed, but is merely representative of selected embodiments of the disclosure. All other embodiments, which can be made by those skilled in the art based on the embodiments of this disclosure without making any inventive effort, are intended to be within the scope of this disclosure.
It should be noted that the term "comprising" will be used in embodiments of the present disclosure to indicate the presence of the stated features hereinafter, but not to preclude the addition of further features.
As shown in fig. 9, in the log storage system in the prior art, a large number of slices are managed on each physical machine, and log records generated by each slice form a log stream. Because each slice is independent, a WAL storage system is deployed for each slice, so that the log stream generated by each slice is additionally written into the log file of the log stream. The use of one log file for each log stream to store logs, respectively, can cause a reduced throughput of the log storage system. In view of the above drawbacks, the present disclosure provides a data processing method and apparatus. The method and the device have the advantages that the logic storage space is respectively arranged for all the log streams in one log file, after the log records of a certain log stream are acquired, the acquired log records are stored in the logic storage space corresponding to the log stream, namely, the fact that the log records corresponding to all the log streams managed by the log storage system are stored in the same log file of a disk is realized, and compared with the fact that the log records corresponding to each log stream managed by the log storage system are respectively stored in one log file in the prior art, the method and the device realize additional writing operation of the log, can effectively improve the log writing efficiency of the log storage system and improve the throughput capacity of the log storage system.
The data processing method and apparatus of the present application are described in detail below by way of specific embodiments.
The embodiment of the disclosure provides a data processing method, which is applied to a terminal device for log storage, and specifically, as shown in fig. 1, may include the following steps:
s110, respectively acquiring a plurality of log records corresponding to each log stream in the log storage system.
Here, each log stream corresponds to a distributed consistency raft instance, that is, the log records generated by each raft instance form a log stream, and the log storage system may be a WAL system. In particular implementations, a journal storage system is deployed on a disk. In this embodiment, a log file is set in a log storage system, and log records corresponding to all log streams are stored in the same log file.
In this step, the log record corresponding to each log stream is obtained, in practical application, one or more log records corresponding to all log streams may be obtained, or only one or more log records corresponding to a part of log streams may be obtained, and for the log streams for which no log record is obtained, the obtained result is null.
S120, determining a logic storage space corresponding to each log stream; wherein, the logic storage space corresponding to all log streams belongs to one log file.
Here, in the log file of the log storage system, a logical storage space for storing log records in the corresponding log stream is set for each log stream, respectively.
In a specific implementation, one shift instance corresponds to one log stream, and this step is to set a logic storage space in a log file of the log storage system for each shift instance, where log records generated by the shift instance are stored in the corresponding logic storage space. All log records generated by the raft instance are stored in the same log file of the log storage system.
The logical storage space is a logically continuous storage space, and is not necessarily a continuous storage area actually existing in the log file, that is, the log file is not blocked by the present disclosure to store log records generated by each raft instance. In the implementation, an identifier is set for the logic storage space corresponding to each raft instance, and meanwhile, the identifier corresponds to the log stream generated by the corresponding raft instance, and the identifier is used for identifying the log record in the log stream generated by the corresponding raft instance. For example, a first identifier is set for a logical storage space corresponding to a certain raft instance, the logical storage space corresponding to the raft instance is identified by the first identifier, and a log record in a log stream generated by the raft instance can identify the first identifier. Thus, the log records generated by the raft instance all identify the first identifier, and after being stored in the log file, all log records with the first identifier belong to the logical storage space corresponding to the first identifier.
S130, storing the plurality of log records corresponding to each log stream into corresponding logic storage spaces respectively.
In step 120 above, a first identifier for identifying the logical storage space of each log stream is set for that log stream; and respectively establishing a mapping relation among each log stream, the first identifier corresponding to each log stream and the logic storage space corresponding to each log stream. In this step, the log records may be stored in the corresponding logical storage space by using the following substeps: acquiring a first identifier corresponding to each log stream; for each log stream, identifying each log record corresponding to the log stream with a first identifier of the log stream.
In practical applications, for example, in a Range or Hash-based distributed key-value storage system, a large number of fragments (Range or Tablet) are managed on each physical machine, all fragments share the same log storage system WalStore, and a plurality of fragments share the same log storage system WalStore essentially write the same log file additionally. As shown in fig. 2, the log records in all the partition-generated log streams WalStream are stored in the corresponding logical storage spaces (logical WALs), respectively, and all the partition-generated log streams are stored in the same log file. When a log file is added to the plurality of fragments together, all the writing to the disk will be sequential writing, and the writing performance of the disk will be fully exerted.
The data processing method of the above embodiment will be described below by way of a specific embodiment. As shown in fig. 3, a log storage system is a WAL system, i.e., a WAL instance, which includes a plurality of log streams WalStream, one WalStream being shared by one raft instance. When a plurality of WalStreams write data, each WalStream writes the data into a Buffer in its own WalStream, then a unified scheduler takes out the data in all buffers and merges them, then adds them into the same log file of the disk to which the log storage system belongs, and builds an index for the data in the memory to accelerate the speed of reading the data.
The Buffer may be identified by using the identifier stream-uuid-N of the corresponding log stream. The log file written with the log is N.log, and the corresponding index is written into the N.index. Because the storage amount of the log files in the disk is preset, one log file cannot be written continuously after being written fully along with the writing, and at the moment, the logs in each log stream WalStream corresponding to all the shift examples can be written into another log file at the same time. After each log file is additionally written, an index is written into a disk. When a Stream reads data, it locates the data according to the file-level index and the block-level index.
When a physical disk is used to store log streams generated by multiple raft instances, each raft instance is written to disk sequentially alone, and ultimately not actually written to disk sequentially. The embodiment achieves real sequential writing to the disk by sharing the same WAL instance by a plurality of raft instances, thereby fully playing the writing performance of the disk. Under the condition of sharing the WAL instance, the WAL instance provides an own exclusive logic storage space for each shift instance, and each shift instance is also sequentially written in the own logic storage space, so that the writing performance of the magnetic disk can be further exerted.
In some embodiments, as shown in FIG. 4, the following steps may be utilized to perform a journaling write operation to fully exploit disk performance, increase the effective data per write, increase the payload ratio:
s410, for each log stream, aggregating the plurality of log records corresponding to the log stream to obtain a log block corresponding to the log stream.
The aggregation processing of the logs specifically aggregates the log records from different log streams, and stores the log records in a single log block, so that the log records are convenient to search and analyze.
As shown in fig. 5, each write thread corresponds to one shift instance, and the write thread stores the log Record generated by the corresponding shift instance into the corresponding Buffer first, and then, aggregates the plurality of log records in each Buffer to obtain three log blocks Span.
S420, storing the log blocks corresponding to the log streams into the log file.
In order to improve the storage efficiency of the log record, the log blocks corresponding to each log stream can be aggregated to obtain a log packet, and the log packet is stored in the log file.
The aggregation processing of the log blocks specifically aggregates the log blocks from different log streams, and stores the log blocks in a single log packet, so that the log blocks are convenient to search and analyze.
As shown in FIG. 5, each log block may be used as a log batch RecordBatch, and the spans are aggregated to obtain a log packet, which may be used as a block batch SpanBatch.
The log batch RecordBatch contains several log records or messages therein, where the messages and log records are equivalent and the log batch and log collection or message collection are equivalent. The block batch span includes a plurality of log blocks.
After obtaining the log package, an index may be set for each log package, and then stored into a log file.
In the embodiment, the log records are gathered for a plurality of times, and then submitted in batches, and the data is sequentially tracked in the log file only by one-time read-write operation, so that the IOPS (Input/Output Operations Per Second) of the disk can be fully improved, and the write performance of the disk is explored by utilizing the bandwidth of the disk. If such a batch commit method is not adopted, each Record is written once, and since the iops of the disk is limited, the data written to the disk per unit time is very small.
In order to implement the mark deletion, i.e. the logical deletion, of the log record in the log storage system, for example, implement the trunk operation of Raft, as shown in fig. 6, the above data processing method may further include the following steps:
s610, determining a second identifier of each log block, and identifying each log block by using the second identifier.
In the above embodiment, the first identifier has been set for each log stream, for example, the first identifier is represented by stream_uuid, and then log records belonging to the same stream_uuid logically belong to the same space. Although a plurality of log records identified by stream_uuid are stored interleaved on a disk, logical space discrimination can be made by stream_uuid. Here, a second identifier index is set for each log block in each log stream, and the second identifier may be used to identify the corresponding log block and each log record in the log block, so that there are multiple log records with the same index in the logical storage space corresponding to the same stream_uuid.
In a specific implementation, the second identifier index is an id that can be provided to the Raft protocol for use, and is a log block corresponding to a certain index that can be rolled back or advanced.
S620, determining a third identifier of each log record based on the generation order of the log in the log stream to which the log belongs.
The multiple log records in the logical storage space corresponding to the same stream_uuid will have the same index, but the logical storage space provided for the shift instance does not allow all log records of the multiple same index to be read at the same time, so this step generates a third identifier seq_id for each log record according to the generation order in the log stream to which the log record belongs, which is a sequence number monotonically increasing by 1, and by recording which seq_ids have been deleted, it can finally be achieved that the data is allowed to be logically deleted in the sequentially written model.
S630, identifying the corresponding log record by using the third identifier.
The above embodiment sets an identifier for each log block and log records in each log block, and using the identifier can logically implement deletion of some logs. Specifically, if the second identifier of the new log block received by the log storage system is smaller than the largest second identifier of the log block corresponding to the log stream to which the new log block belongs, which is stored in the log storage system, at this time, the log block in the log storage system may be marked for deletion or logically deleted by:
determining a target log stream to which the new log block belongs; determining a largest second identifier in a log block corresponding to the target log stream stored in the log storage system; and setting the log blocks which are stored in the log storage system and correspond to the target log stream and have the second identifiers larger than or equal to the second identifiers of the new log blocks as invalid states.
In a specific application, a certain machine finds itself not as a host, but the second identifier of the log block in the target log stream stored in the host is smaller than the second identifier of the log block in the target log stream stored in the machine, at this time, the machine needs to set the log block in which the second identifier in the target log stream stored in the machine is greater than or equal to the second identifier of the newly received log block to be in a failure state, and start from the second identifier of the newly received log block, perform log overwriting.
If the second identifier of the new log block received by the log storage system is greater than the largest second identifier of the log block corresponding to the log stream to which the new log block belongs, which is stored in the log storage system, at this time, the log block in the log storage system may be marked for deletion or logically deleted by using the following steps:
determining a target log stream to which the new log block belongs; and setting the log block corresponding to the target log stream stored in the log storage system as a failure state.
Corresponding to the data processing method, the embodiment of the disclosure further provides a data processing device, where the device is applied to a terminal device that stores and processes logs, and the device and its respective modules can execute the same method steps as the data processing method, and achieve the same or similar beneficial effects, so that repeated parts are not repeated.
As shown in fig. 7, the data processing apparatus provided by the present disclosure includes:
the log obtaining module 710 is configured to obtain a plurality of log records corresponding to each log stream in the log storage system respectively;
a storage determining module 720, configured to determine a logical storage space corresponding to each log stream; wherein, the logic storage space corresponding to all log streams belongs to one log file;
and a storage module 730, configured to store the plurality of log records corresponding to each log stream into a corresponding logical storage space.
In some embodiments, the storage determining module 720 is configured, when determining the logical storage space corresponding to each log stream, to:
setting a first identifier for identifying a logic storage space of each log stream for the log stream respectively;
respectively establishing a mapping relation among each log stream, a first identifier corresponding to each log stream and a logic storage space corresponding to each log stream;
the storage module 730 is configured to, when storing the plurality of log records corresponding to each log stream into a corresponding logical storage space, respectively:
acquiring a first identifier corresponding to each log stream;
for each log stream, identifying each log record corresponding to the log stream with a first identifier of the log stream.
The embodiment of the disclosure discloses an electronic device, as shown in fig. 8, including: a processor 801, a memory 802, and a bus 803, the memory 802 storing machine readable instructions executable by the processor 801, the processor 801 and the memory 802 communicating via the bus 803 when the electronic device is operating.
The machine readable instructions, when executed by the processor 801, perform the steps of a data processing method of:
respectively acquiring a plurality of log records corresponding to each log stream in a log storage system;
determining a logic storage space corresponding to each log stream; wherein, the logic storage space corresponding to all log streams belongs to one log file;
and respectively storing the plurality of log records corresponding to each log stream into a corresponding logic storage space.
In addition, when the machine-readable instructions are executed by the processor 801, the method content in any of the embodiments described in the method section above may be executed, and will not be described in detail herein.
The embodiment of the disclosure also provides a computer program product corresponding to the above method and apparatus, including a computer readable storage medium storing program code, where instructions included in the program code may be used to execute the method in the foregoing method embodiment, and specific implementation may refer to the method embodiment and will not be described herein.
The foregoing description of various embodiments is intended to highlight differences between the various embodiments, which may be the same or similar to each other by reference, and is not repeated herein for the sake of brevity.
It will be clearly understood by those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described system and apparatus may refer to corresponding procedures in the method embodiments, and will not be described in detail in this disclosure. In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. The above-described apparatus embodiments are merely illustrative, and the division of the modules is merely a logical function division, and there may be additional divisions when actually implemented, and for example, multiple modules or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some communication interface, indirect coupling or communication connection of devices or modules, electrical, mechanical, or other form.
The modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical units, may be located in one place, or may be distributed over multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in each embodiment of the present disclosure may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer readable storage medium executable by a processor. Based on such understanding, the technical solution of the present disclosure may be embodied in essence or a part contributing to the prior art or a part of the technical solution, or in the form of a software product stored in a storage medium, including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in the embodiments of the present disclosure. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk, etc.
The foregoing is merely a specific embodiment of the disclosure, but the protection scope of the disclosure is not limited thereto, and any person skilled in the art can easily think about changes or substitutions within the technical scope of the disclosure, and it should be covered in the protection scope of the disclosure. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.
Claims (8)
1. A method of data processing, comprising:
respectively acquiring a plurality of log records corresponding to each log stream in a log storage system;
determining a logic storage space corresponding to each log stream; wherein, the logic storage space corresponding to all log streams belongs to one log file;
for each log stream, aggregating the plurality of log records corresponding to the log stream to obtain a log block corresponding to the log stream;
storing the log blocks corresponding to each log stream into the log file;
determining a second identifier for each log block and identifying each log block with the second identifier;
receiving a new log block to be written;
determining a target log stream to which the new log block belongs;
determining a largest second identifier in a log block corresponding to the target log stream stored in the log storage system;
if the second identifier of the new log block is smaller than the maximum second identifier, setting the log block, of which the second identifier is larger than or equal to the second identifier of the new log block, in the log blocks corresponding to the target log stream stored in the log storage system as an invalid state; or alternatively, the process may be performed,
and if the second identifier of the new log block is larger than the maximum second identifier, setting the log block corresponding to the target log stream stored in the log storage system as a failure state.
2. The method of claim 1, wherein determining the logical storage space corresponding to each log stream comprises:
setting a first identifier for identifying a logic storage space of each log stream for the log stream respectively;
respectively establishing a mapping relation among each log stream, a first identifier corresponding to each log stream and a logic storage space corresponding to each log stream;
the storing the log blocks corresponding to the log streams in the log file includes:
acquiring a first identifier corresponding to each log stream;
for each log stream, identifying each log record corresponding to the log stream with a first identifier of the log stream.
3. The method according to claim 1, wherein storing the log blocks corresponding to the respective log streams in the log file includes:
carrying out aggregation processing on the log blocks corresponding to each log stream to obtain a log packet;
and storing the log package into the log file.
4. The data processing method according to claim 1, characterized by further comprising:
for each log record, determining a third identifier of the log record based on an order of generation of the log record in a log stream to which the log record belongs;
and identifying the corresponding log record by using the third identifier.
5. A data processing apparatus, comprising:
the log acquisition module is used for respectively acquiring a plurality of log records corresponding to each log stream in the log storage system;
the storage determining module is used for determining a logic storage space corresponding to each log stream; wherein, the logic storage space corresponding to all log streams belongs to one log file;
the storage module is used for carrying out aggregation processing on the plurality of log records corresponding to each log stream to obtain a log block corresponding to the log stream; storing the log blocks corresponding to each log stream into the log file; determining a second identifier for each log block and identifying each log block with the second identifier; receiving a new log block to be written; determining a target log stream to which the new log block belongs; determining a largest second identifier in a log block corresponding to the target log stream stored in the log storage system; if the second identifier of the new log block is smaller than the maximum second identifier, setting the log block, of which the second identifier is larger than or equal to the second identifier of the new log block, in the log blocks corresponding to the target log stream stored in the log storage system as an invalid state; or if the second identifier of the new log block is greater than the maximum second identifier, setting the log block corresponding to the target log stream stored in the log storage system to be in a failure state.
6. The data processing apparatus of claim 5, wherein the storage determination module, when determining the logical storage space corresponding to each log stream, is configured to:
setting a first identifier for identifying a logic storage space of each log stream for the log stream respectively;
respectively establishing a mapping relation among each log stream, a first identifier corresponding to each log stream and a logic storage space corresponding to each log stream;
the storage module is used for storing the log blocks corresponding to the log streams into the log file when the log blocks are stored in the log file:
acquiring a first identifier corresponding to each log stream;
for each log stream, identifying each log record corresponding to the log stream with a first identifier of the log stream.
7. An electronic device, comprising: the system comprises a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, when the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to execute the data processing method according to any one of claims 1-4.
8. A computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, performs the data processing method according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010348381.8A CN111563017B (en) | 2020-04-28 | 2020-04-28 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010348381.8A CN111563017B (en) | 2020-04-28 | 2020-04-28 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111563017A CN111563017A (en) | 2020-08-21 |
| CN111563017B true CN111563017B (en) | 2023-05-16 |
Family
ID=72071728
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010348381.8A Active CN111563017B (en) | 2020-04-28 | 2020-04-28 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111563017B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112486777A (en) * | 2020-12-11 | 2021-03-12 | 深圳前瞻资讯股份有限公司 | Big data service program log processing method and system |
| CN115185787B (en) * | 2022-09-06 | 2022-12-30 | 北京奥星贝斯科技有限公司 | Method and device for processing transaction log |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107408070A (en) * | 2014-12-12 | 2017-11-28 | 微软技术许可有限责任公司 | More transaction journals in distributed memory system |
| CN109446174A (en) * | 2018-10-30 | 2019-03-08 | 东软集团股份有限公司 | Logdata record method, apparatus and computer readable storage medium |
| CN109508144A (en) * | 2018-08-30 | 2019-03-22 | 郑州云海信息技术有限公司 | A kind of log processing method and relevant apparatus |
| CN109542733A (en) * | 2018-12-05 | 2019-03-29 | 焦点科技股份有限公司 | A kind of highly reliable real-time logs collection and visual m odeling technique method |
| CN110209642A (en) * | 2018-02-05 | 2019-09-06 | 北京智明星通科技股份有限公司 | Method, apparatus, server and the computer-readable medium of information processing |
| WO2020060621A1 (en) * | 2018-09-21 | 2020-03-26 | Microsoft Technology Licensing, Llc | Log destaging from fixed-size log portion |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130332435A1 (en) * | 2012-06-12 | 2013-12-12 | Microsoft Corporation | Partitioning optimistic concurrency control and logging |
| US10528404B2 (en) * | 2018-06-01 | 2020-01-07 | Micron Technology, Inc. | Event logging in a multi-core system |
| CN109508246A (en) * | 2018-06-25 | 2019-03-22 | 广州多益网络股份有限公司 | Log recording method, system and computer readable storage medium |
| CN109347899B (en) * | 2018-08-22 | 2022-03-25 | 北京百度网讯科技有限公司 | Method for writing log data in distributed storage system |
-
2020
- 2020-04-28 CN CN202010348381.8A patent/CN111563017B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107408070A (en) * | 2014-12-12 | 2017-11-28 | 微软技术许可有限责任公司 | More transaction journals in distributed memory system |
| CN110209642A (en) * | 2018-02-05 | 2019-09-06 | 北京智明星通科技股份有限公司 | Method, apparatus, server and the computer-readable medium of information processing |
| CN109508144A (en) * | 2018-08-30 | 2019-03-22 | 郑州云海信息技术有限公司 | A kind of log processing method and relevant apparatus |
| WO2020060621A1 (en) * | 2018-09-21 | 2020-03-26 | Microsoft Technology Licensing, Llc | Log destaging from fixed-size log portion |
| CN109446174A (en) * | 2018-10-30 | 2019-03-08 | 东软集团股份有限公司 | Logdata record method, apparatus and computer readable storage medium |
| CN109542733A (en) * | 2018-12-05 | 2019-03-29 | 焦点科技股份有限公司 | A kind of highly reliable real-time logs collection and visual m odeling technique method |
Non-Patent Citations (1)
| Title |
|---|
| 基于分布式流计算的运维安全分析;吴志力;陈希;杨世登;;网络安全技术与应用(第02期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111563017A (en) | 2020-08-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106406758B (en) | A kind of data processing method and storage equipment based on distributed memory system | |
| CN108255647B (en) | High-speed data backup method under samba server cluster | |
| US9298707B1 (en) | Efficient data storage and retrieval for backup systems | |
| US7890716B2 (en) | Method of managing time-based differential snapshot | |
| US11232073B2 (en) | Method and apparatus for file compaction in key-value store system | |
| US9785514B1 (en) | Techniques for file system recovery | |
| CN111563017B (en) | Data processing method and device | |
| CN103516736A (en) | Data recovery method of distributed cache system and a data recovery device of distributed cache system | |
| US20120047115A1 (en) | Extent reference count update system and method | |
| US10091293B2 (en) | Rapid cloud-based image centralization | |
| CN103765373A (en) | Data storage method, data storage device, and storage equipment | |
| CN112905390A (en) | Log data backup method, device, equipment and storage medium | |
| CN104965835B (en) | A kind of file read/write method and device of distributed file system | |
| US10592139B2 (en) | Embedded object data storage determined by object size information | |
| EP3229138B1 (en) | Method and device for data backup in a storage system | |
| CN104583966A (en) | Backup and restore system for a deduplicated file system and corresponding server and method | |
| CN113885809B (en) | Data management system and method | |
| CN110121694A (en) | A kind of blog management method, server and Database Systems | |
| US20220342851A1 (en) | File system event monitoring using metadata snapshots | |
| CN111857603B (en) | Data processing method and related device | |
| CN105068760B (en) | Date storage method, data storage device and storage device | |
| CN103399714B (en) | The maintenance system of storage device configurations information and using method thereof | |
| CN115454717B (en) | Database real-time backup method and device, computer equipment and storage medium | |
| CN112069162B (en) | Data processing method and device for stream computation, computer equipment and storage medium | |
| CN111399774B (en) | Data processing method and device based on snapshot under distributed storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP01 | Change in the name or title of a patent holder | ||
| CP01 | Change in the name or title of a patent holder |
Address after: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Patentee after: Douyin Vision Co.,Ltd. Address before: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Patentee before: Tiktok vision (Beijing) Co.,Ltd. Address after: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Patentee after: Tiktok vision (Beijing) Co.,Ltd. Address before: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Patentee before: BEIJING BYTEDANCE NETWORK TECHNOLOGY Co.,Ltd. |