CN116483843A - Data processing method, device, electronic equipment and storage medium - Google Patents

Data processing method, device, electronic equipment and storage medium Download PDF

Info

Publication number
CN116483843A
CN116483843A CN202210039129.8A CN202210039129A CN116483843A CN 116483843 A CN116483843 A CN 116483843A CN 202210039129 A CN202210039129 A CN 202210039129A CN 116483843 A CN116483843 A CN 116483843A
Authority
CN
China
Prior art keywords
data
target
block
consensus
group
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210039129.8A
Other languages
Chinese (zh)
Inventor
张作青
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Kingsoft Cloud Network Technology Co Ltd
Original Assignee
Beijing Kingsoft Cloud Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Kingsoft Cloud Network Technology Co Ltd filed Critical Beijing Kingsoft Cloud Network Technology Co Ltd
Priority to CN202210039129.8A priority Critical patent/CN116483843A/en
Publication of CN116483843A publication Critical patent/CN116483843A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G06F16/2365—Ensuring data consistency and integrity
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/182—Distributed file systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/064—Management of blocks
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Computer Security & Cryptography (AREA)
  • Human Computer Interaction (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The embodiment of the invention provides a data processing method, a device, electronic equipment and a storage medium, wherein the method comprises the following steps: under the condition that a data modification instruction sent by a client is received, analyzing original data carried in the data modification instruction; searching a target consensus group from a preset consensus group set, wherein the original data are locally stored in a target main block and a target slave block in the target consensus group; the method comprises the steps of selecting a master block and a slave block from a plurality of preset block groups respectively through a preset consensus algorithm to obtain a plurality of consensus groups, wherein the consensus groups form a consensus group set; and sending the data modification instruction to the target main block so that the target main block executes the data modification instruction to modify the original data, and forwarding the data modification instruction to the target slave block so that the target slave block executes the data modification instruction to modify the original data. Thus, a consensus algorithm is adopted to ensure data consistency.

Description

Data processing method, device, electronic equipment and storage medium
Technical Field
The present invention relates to the field of internet technologies, and in particular, to a data processing method, a data processing device, an electronic device, and a storage medium.
Background
A distributed file storage system is typically composed of three parts: client, proxy server and storage. The client is responsible for receiving all data operation instructions of the user, the proxy server is responsible for authentication, and forwarding all data operation instructions of the user to a corresponding storage end, wherein the storage end is usually composed of N storage servers, and the storage end stores data of the user.
Wherein each storage server is physically divided into a plurality of fixed-length blocks (blocks), each block having a unique identification (block id). A group is typically formed by blocks on different storage servers, where the blocks in the group store data that is completely consistent, thereby enabling backup of the data. For example, a group is made up of 3 blocks, the 3 blocks being located on different storage servers, the data of the 3 blocks being completely identical.
Currently, for the scenario of modifying data in a distributed file storage system, when modifying certain data, if a certain block of data in a group is successfully modified, for example, data a is modified to a', that is, the execution of a data modification instruction is considered to be successful, however, the actual situation may be that the modification of the rest blocks of data in the group fails, so that the data consistency problem cannot be guaranteed.
Disclosure of Invention
The embodiment of the invention aims to provide a data processing method, a data processing device, electronic equipment and a storage medium, so as to realize the beneficial effect of guaranteeing the data consistency. The specific technical scheme is as follows:
in a first aspect of the embodiment of the present invention, there is first provided a data processing method, including:
under the condition that a data modification instruction sent by a client is received, analyzing original data carried in the data modification instruction;
searching a target consensus group from a preset consensus group set, wherein the original data are locally stored in a target main block and a target slave block in the target consensus group;
the method comprises the steps of selecting a master block and a slave block from a plurality of preset block groups respectively through a preset consensus algorithm to obtain a plurality of consensus groups, wherein the consensus groups form a consensus group set;
and sending the data modification instruction to the target main block so that the target main block executes the data modification instruction to modify the original data, and forwarding the data modification instruction to the target slave block so that the target slave block executes the data modification instruction to modify the original data.
In an optional embodiment, the searching the target consensus group from the preset consensus group set includes:
Analyzing target data carried in the data modification instruction, wherein the target data comprises data modified for the original data;
acquiring a data modification quantity threshold value set for a data modification scene, and judging whether the data quantity of the target data is smaller than or equal to the data modification quantity threshold value;
if the data quantity of the target data is smaller than or equal to the data modification quantity threshold value, searching a target consensus group from a preset consensus group set.
In an optional embodiment, the searching the target consensus group from the preset consensus group set includes:
analyzing target data carried in the data modification instruction, wherein the target data comprises data modified for the original data;
acquiring a data grade set for a data modification scene, and judging whether the grade of the target data is greater than or equal to the data grade;
if the grade of the target data is greater than or equal to the grade of the data, searching a target consensus group from a preset consensus group set.
In an alternative embodiment, the method further comprises:
receiving data modification success messages returned by the target main block and the target slave block, and counting the number of the data modification success messages;
Judging whether the number of the data modification success messages is larger than or equal to a preset message number threshold value, wherein the preset message number threshold value is smaller than or equal to the number of elements in the target consensus group;
if the number of the data modification success messages is larger than or equal to the preset message number threshold, returning a message that the data modification instruction is successfully executed to the client;
and if the number of the data modification success messages is smaller than the preset message number threshold, returning a message of failure in executing the data modification instruction to the client.
In an alternative embodiment, the method further comprises:
if the data quantity of the target data is larger than the data modification quantity threshold value, searching a target block group from a plurality of block groups, wherein the blocks in the target block group locally store the original data;
selecting a target block from blocks included in the target block group, and reading the original data from the target block;
and modifying the original data according to the target data, and writing the modified original data into the blocks included in the target block group.
In an alternative embodiment, the method further comprises:
If the grade of the target data is smaller than the grade of the data, searching a target block group from a plurality of block groups, wherein the blocks in the target block group all store the original data;
selecting a target block from blocks included in the target block group, and reading the original data from the target block;
and modifying the original data according to the target data, and writing the modified original data into the blocks included in the target block group.
In an optional embodiment, the selecting a target block from the blocks included in the target block group includes:
acquiring a block selection threshold set for a data modification scene, and judging whether the data amount of the target data is smaller than or equal to the block selection threshold;
and if the data quantity of the target data is smaller than or equal to the block selection threshold value, randomly selecting a target block from the blocks included in the target block group according to a preset random selection rule.
In an optional embodiment, the selecting a target block from the blocks included in the target block group further includes:
if the data volume of the target data is larger than the block selection threshold, determining a storage server where each block in the target block group is located, and acquiring performance parameters of the storage server;
And randomly selecting a target block from the blocks of the target block group according to the performance parameters of the storage server.
In an alternative embodiment, the performance parameter includes at least one of: memory size, network conditions, CPU performance;
the selecting, according to the performance parameter of the storage server, a target block randomly from the blocks in the target block group includes:
acquiring election parameters corresponding to the memory size, the network condition and the CPU performance of the storage server;
and acquiring the sum of the election parameters, or acquiring a weighted sum of the election parameters, and selecting a target block from the blocks of the target block group according to the sum or the weighted sum.
In a second aspect of the embodiments of the present invention, there is also provided a data processing apparatus, the apparatus including:
the data analysis module is used for analyzing the original data carried in the data modification instruction under the condition that the data modification instruction sent by the client is received;
the consensus group searching module is used for searching a target consensus group from a preset consensus group set, wherein the target main block and the target slave block in the target consensus group are locally stored with the original data;
The method comprises the steps of selecting a master block and a slave block from a plurality of preset block groups respectively through a preset consensus algorithm to obtain a plurality of consensus groups, wherein the consensus groups form a consensus group set;
and the data processing module is used for sending the data modification instruction to the target main block so that the target main block executes the data modification instruction to modify the original data, and forwarding the data modification instruction to the target slave block so that the target slave block executes the data modification instruction to modify the original data.
In a third aspect of the embodiment of the present invention, there is also provided an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus;
a memory for storing a computer program;
and a processor, configured to implement the data processing method according to any one of the first aspect when executing the program stored in the memory.
In a fourth aspect of embodiments of the present invention, there is also provided a storage medium having stored therein instructions which, when executed on a computer, cause the computer to perform the data processing method of any of the first aspects described above.
In a fifth aspect of embodiments of the present invention, there is also provided a computer program product comprising instructions which, when run on a computer, cause the computer to perform the data processing method of any of the first aspects described above.
According to the technical scheme provided by the embodiment of the invention, under the condition that the data modification instruction sent by the client is received, the original data carried in the data modification instruction is analyzed, the target consensus group is searched from the preset consensus group set, wherein the original data is locally stored in the target main block and the target slave block in the target consensus group, the main block and the slave block are respectively elected from the preset multiple block groups through the preset consensus algorithm to obtain multiple consensus groups, the multiple consensus groups form the consensus group set, the data modification instruction is sent to the target main block, so that the target main block executes the data modification instruction to modify the original data, and the data modification instruction is transferred to the target slave block, so that the target slave block executes the data modification instruction to modify the original data. And respectively selecting a master block and a slave block from a plurality of preset block groups through a preset consensus algorithm to obtain a plurality of consensus groups, forming a consensus group set by the plurality of consensus groups, subsequently searching a target consensus group from the preset consensus group set, sending a data modification instruction to a target master block in the target consensus group so as to enable the target master block to execute the data modification instruction to modify original data, and forwarding the data modification instruction to a target slave block in the target consensus group by the target master block so as to enable the target slave block to execute the data modification instruction to modify the original data, thereby ensuring data consistency by adopting the consensus algorithm.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
In order to more clearly illustrate the embodiments of the invention or the technical solutions of the prior art, the drawings which are used in the description of the embodiments or the prior art will be briefly described, and it will be obvious to a person skilled in the art that other drawings can be obtained from these drawings without inventive effort.
FIG. 1 is a schematic diagram of an implementation flow of a data processing method according to an embodiment of the present invention;
fig. 2 is a schematic diagram of an application scenario in an embodiment of the present invention;
FIG. 3 is a flow chart illustrating another data processing method according to an embodiment of the present invention;
FIG. 4 is a flow chart illustrating another data processing method according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a data processing apparatus according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
As shown in fig. 1, a schematic implementation flow chart of a data processing method provided by an embodiment of the present invention is executed by a proxy server, and may specifically include the following steps:
s101, under the condition that a data modification instruction sent by a client is received, analyzing original data carried in the data modification instruction.
In the embodiment of the invention, a user can submit all data operation instructions at a client, wherein the data operation instructions can comprise data modification, storage, viewing and other instructions. In the data modification scene, a user can submit a data modification instruction at a client, so that the client can receive the data modification instruction submitted by the user and send the data modification instruction to the proxy server.
For the proxy server, the data modification instruction sent by the client can be received, and the original data carried in the data modification instruction can be analyzed under the condition that the data modification instruction sent by the client is received. In addition, the data modification instruction carries target data, which is the data modified from the original data.
For example, as shown in fig. 2, for the proxy server, the data modification instruction sent by the client may be received, and in the case that the data modification instruction sent by the client is received, the original data a carried in the data modification instruction is parsed, and in addition, the data modification instruction carries the target data a ', that is, the target data a' is obtained after the original data a is modified.
S102, searching a target consensus group from a preset consensus group set, wherein the target main block and the target slave block in the target consensus group are locally stored with the original data.
In the embodiment of the present invention, a storage end (the storage end is typically composed of N storage servers) is divided into a plurality of block groups in advance, each block group includes a plurality of blocks, and the blocks included in each block group are located in different storage servers. For example, each block group includes 3 blocks: block 1, block 2, block 3, and the blocks included in each block group are located on different storage servers, i.e. block 1 is located on storage server 1, block 2 is located on storage server 2, and block 3 is located on storage server 3.
For the plurality of block groups, in the embodiment of the present invention, a master block (leader) and a slave block (follower) are selected from the plurality of block groups through a preset consensus algorithm, so that a plurality of consensus groups can be obtained, and a consensus group set is formed by the plurality of consensus groups. For example, for block group one includes 3 blocks: block 1, block 2, block 3, from which one master block (leader) and two slaves (follower) are elected by a consensus algorithm, after which services can be provided externally.
It should be noted that, for the consensus algorithm, the shift algorithm may be specifically used, and of course, other consensus algorithms may also be used, which is not limited in the embodiment of the present invention. In this way, a master block and a slave block can be respectively selected from a plurality of preset block groups through a preset consensus algorithm, a plurality of consensus groups are obtained, and the consensus group set is formed by a plurality of the consensus groups.
Based on this, in the embodiment of the present invention, the target consensus group may be searched from a preset consensus group set, where both the target master block and the target slave block in the target consensus group locally store the original data, which means that the target consensus group storing the original data is searched from the preset consensus group set. For example, assume that the consensus set includes consensus set 1, consensus set 2, and consensus set 3, from which consensus set 1 is found, one master block (leader) and two slave blocks (follower) in consensus set 1 each store the original data a locally.
The data modification instruction generally carries target data, and the embodiment of the invention analyzes the target data carried in the data modification instruction, wherein the target data comprises data modified aiming at original data. For example, the original data a, the data a' modified from the original data a is the target data. Based on the data size of the target data, it is determined whether to execute the above step S102, i.e., whether to select a data processing mode with better I/O performance.
Specifically, the embodiment of the invention acquires a data modification amount threshold set for a data modification scene, judges whether the data amount of target data is smaller than or equal to the data modification amount threshold (used for determining whether to select a data processing mode with better I/O performance), and searches a target consensus group from a preset consensus group set if the data amount of the target data is smaller than or equal to the data modification amount threshold.
For example, in the embodiment of the present invention, a data modification amount threshold is preset, and if the data modification amount threshold is 2KB, and the data amount of the data is smaller than or equal to the value, a data processing mode with better I/O performance may be selected, based on this, it is determined whether the data amount of the target data (e.g., a') is smaller than or equal to 2KB, and if the data amount of the target data is smaller than or equal to 2KB, the target consensus group is searched from the preset consensus group set.
In addition, in view of the fact that data generally has a priority attribute, it is considered that the higher the priority, the more important the data, and the higher the I/O performance requirement is, based on this, the embodiment of the present invention may determine whether to execute the above step S102 based on the importance degree of the data, that is, determine whether to select a data processing mode with better I/O performance.
Specifically, in the embodiment of the present invention, a data level set for a data modification scene is obtained (used for determining whether to select a data processing mode with better I/O performance), and whether the level of target data is greater than or equal to the data level is determined, if the level of target data is greater than or equal to the data level, the target consensus group is searched from a preset consensus group set.
For example, in the embodiment of the present invention, a data level is preset, and it is assumed to be 2 levels (the higher the level is, the more important the data is), which means that the data level is greater than or equal to the data level, and a data processing mode with better I/O performance may be selected, based on this, the level of the target data (e.g., a') (i.e., the data level is 3 levels), so that it may be determined that the level of the target data is greater than 2 levels, and then the target consensus group is searched from the preset consensus group set.
And S103, the data modification instruction is sent to the target main block, so that the target main block executes the data modification instruction to modify the original data, and the data modification instruction is forwarded to the target slave block, so that the target slave block executes the data modification instruction to modify the original data.
In the embodiment of the invention, the data modification instruction can be (directly) sent to the target main block (i.e. the leader), so that the target main block (i.e. the leader) can receive the data modification instruction, and further can execute the data modification instruction to modify the original data, thereby the target main block (i.e. the leader) finishes modification of the original data.
In addition, for the target master block (i.e., leader), the data modification instruction may be forwarded to the target slave block (i.e., follower), so that the target slave block (i.e., follower) may execute the data modification instruction to modify the original data, so that the target slave block (i.e., follower) completes modification of the original data, and thus, data consistency is ensured through the consensus algorithm.
For the target slave block (i.e., the loader), if the execution of the data modification instruction is successful, a data modification success message may be returned to the target master block (i.e., the loader), and then the target master block (i.e., the loader) may replace the target slave block (i.e., the loader) to return the data modification success message, while for the target master block (i.e., the loader), if the execution of the data modification instruction is successful, a data modification success message may be returned as well.
Based on the above, the embodiment of the invention can receive the data modification success information returned by the target main block and the target slave block, count the number of the data modification success information, judge whether the number of the data modification success information is larger than or equal to the preset information number threshold, the preset information number threshold is smaller than or equal to the number of elements in the target consensus group, if the number of the data modification success information is larger than or equal to the preset information number threshold, return the information that the data modification instruction is successfully executed to the client, and if the number of the data modification success information is smaller than the preset information number threshold, return the information that the data modification instruction is failed to be executed to the client.
For example, the embodiment of the present invention receives the data modification success message returned by the target master block (i.e., leader) and the target slave block (i.e., follower), counts the number of the data modification success messages, and determines whether the number of the data modification success messages is greater than or equal to the preset message number threshold (2), where the target consensus group includes 1 target master block (i.e., leader) and 2 target slave blocks (i.e., follower), meaning that the number of elements in the target consensus group is 3, meaning that the preset message number threshold is less than the number of elements in the target consensus group.
If the number of the data modification success messages is greater than or equal to a preset message number threshold (2), indicating that the data modification instructions of most elements in the target consensus group are successful in execution, determining that the data modification instructions are successful in execution at the moment, and returning the message of successful execution of the data modification instructions to the client; if the number of the data modification success messages is smaller than the preset message number threshold (2), the data modification instruction execution of a very small part of elements in the target consensus group or not successful in the elements is indicated, at this time, the data modification instruction execution failure can be determined, and a message of the data modification instruction execution failure is returned to the client.
Through the description of the technical scheme provided by the embodiment of the invention, under the condition that the data modification instruction sent by the client is received, the original data carried in the data modification instruction is analyzed, the target consensus group is searched from the preset consensus group set, wherein the target main block and the target slave block in the target consensus group are both locally stored with the original data, the main block and the slave block are respectively selected from the preset block groups through the preset consensus algorithm to obtain a plurality of consensus groups, the consensus group set is formed by the plurality of consensus groups, the data modification instruction is sent to the target main block, so that the target main block executes the data modification instruction to modify the original data, and the data modification instruction is transferred to the target slave block, so that the target slave block executes the data modification instruction to modify the original data.
And respectively selecting a master block and a slave block from a plurality of preset block groups through a preset consensus algorithm to obtain a plurality of consensus groups, forming a consensus group set by the plurality of consensus groups, subsequently searching a target consensus group from the preset consensus group set, sending a data modification instruction to a target master block in the target consensus group so as to enable the target master block to execute the data modification instruction to modify original data, and forwarding the data modification instruction to a target slave block in the target consensus group by the target master block so as to enable the target slave block to execute the data modification instruction to modify the original data, thereby ensuring data consistency by adopting the consensus algorithm.
As shown in fig. 3, a schematic implementation flow chart of another data processing method provided by an embodiment of the present invention is executed by a proxy server, and may specifically include the following steps:
s301, if the data volume of the target data is larger than the data modification volume threshold, searching a target block group from a plurality of block groups, wherein the blocks in the target block group are locally stored with the original data.
In the embodiment of the invention, for the data volume of the target data, if the data volume of the target data is larger than the data modification quantity threshold value, the data volume is larger, at the moment, a data processing mode with general I/O performance can be selected, the pressure of a storage end is relieved, and the storage end is prevented from collapsing, so that the target block group can be searched from a plurality of block groups, wherein the blocks in the target block group locally store the original data.
For example, in the embodiment of the present invention, a data modification amount threshold is preset, and assuming that the data modification amount is 2KB, and the data amount of the data is greater than the threshold, a data processing manner with general I/O performance may be selected, and based on this, a target block group may be searched from a plurality of block groups, where all blocks in the target block group locally store the original data a.
S302, selecting a target block from blocks included in the target block group, and reading the original data from the target block.
For a target block group, which generally includes a plurality of blocks, embodiments of the present invention select a target block from among the blocks included in the target block group, and read the original data from the target block.
For example, for a target block group, including: block 1, block 2, block 3, the embodiment of the present invention selects block 1 from the blocks included in the target block group as the target block, and reads the original data a from the target block.
S303, modifying the original data according to the target data, and writing the modified original data into the blocks included in the target block group.
In the embodiment of the invention, the original data can be modified according to the target data in the cache, and the modified original data is (again) written into the blocks included in the target block group, so that the whole data processing flow is subjected to one-time data reading and writing, and the I/O performance is general, but the data consistency can be ensured.
For example, in the embodiment of the present invention, the original data a is modified according to the target data a ' in the memory, so that the original data a is modified into the target data a ', and then the modified original data (i.e., a ') can be written into the block 1, the block 2 and the block 3 again, so that the consistency of the data stored in all the blocks in the target block group can be ensured, and the problem of data consistency is avoided.
As shown in fig. 4, a schematic implementation flow chart of another data processing method provided by an embodiment of the present invention is executed by a proxy server, and may specifically include the following steps:
s401, if the grade of the target data is smaller than the grade of the data, searching a target block group from a plurality of block groups, wherein the blocks in the target block group all store the original data.
In the embodiment of the invention, for the grade of the target data, if the grade of the target data is smaller than the grade of the data, the priority of the target data is not high, and the importance degree is general, at the moment, a data processing mode with general I/O performance can be selected, the pressure of a storage end is relieved, and the storage end is prevented from collapsing, so that the target block group can be searched from a plurality of block groups, wherein the blocks in the target block group locally store the original data.
For example, in the embodiment of the present invention, a data level is preset, and it is assumed to be level 2 (the higher the level is, the more important the data is), which means that the level of the data is smaller than the data level, and a data processing mode with general I/O performance may be selected, based on which a target block group may be searched from a plurality of block groups, where all blocks in the target block group locally store the original data a.
S402, selecting a target block from blocks included in the target block group, and reading the original data from the target block.
For a target block group, which generally includes a plurality of blocks, embodiments of the present invention select a target block from among the blocks included in the target block group, and read the original data from the target block.
For example, for a target block group, including: block 1, block 2, block 3, the embodiment of the present invention selects block 1 from the blocks included in the target block group as the target block, and reads the original data a from the target block.
S403, modifying the original data according to the target data, and writing the modified original data into the blocks included in the target block group.
In the embodiment of the invention, the original data can be modified according to the target data in the cache, and the modified original data is (again) written into the blocks included in the target block group, so that the whole data processing flow is subjected to one-time data reading and writing, and the I/O performance is general, but the data consistency can be ensured.
For example, in the embodiment of the present invention, the original data a is modified according to the target data a ' in the memory, so that the original data a is modified into the target data a ', and then the modified original data (i.e., a ') can be written into the block 1, the block 2 and the block 3 again, so that the consistency of the data stored in all the blocks in the target block group can be ensured, and the problem of data consistency is avoided.
In addition, in the embodiment of the present invention, the target block may be specifically selected from the blocks included in the target block group by: and acquiring a block selection threshold set for the data modification scene, judging whether the data volume of the target data is smaller than or equal to the block selection threshold, and randomly selecting a target block from the blocks included in the target block group according to a preset random selection rule if the data volume of the target data is smaller than or equal to the block selection threshold.
For example, in the embodiment of the present invention, a block selection threshold is set in advance for a data modification scene, assuming 0.5KB, whether the data size of the target data is less than or equal to 0.5KB is determined, if the data size of the target data is less than or equal to 0.5KB, it is not important for selecting which block to read the original data, and at this time, the target block may be randomly selected from the blocks included in the target block group according to a preset random selection rule.
In addition, if the data volume of the target data is larger than the block selection threshold, indicating that the modification volume of the data is larger, selecting a better block to read the original data, so as to determine the storage server where each block in the target block group is located, and acquiring the performance parameters of the storage server; and selecting the target block from the blocks in the target block group according to the performance parameters of the storage server.
For example, if the data size of the target data is greater than 0.5KB, which indicates that the modification amount of the data is greater, a better block may be selected to read the original data, so that the storage server where each block in the target block group is located is determined, and the performance parameters of the storage server are obtained; and selecting the target block from the blocks in the target block group according to the performance parameters of the storage server.
Specifically, in the embodiment of the present invention, the performance parameters include at least one of the following: the memory size, the network condition and the CPU performance can be obtained, the election parameters corresponding to the memory size, the network condition and the CPU performance of the storage server can be obtained, the sum of the election parameters is obtained, or the weighted sum of the election parameters is obtained, and the target block is selected from the blocks of the target block group according to the sum or the weighted sum.
The memory size may be generally understood as the capacity of the content, such as 8GB memory, 16GB memory, for network conditions, network latency, CPU performance, CPU model, parameters, etc., which is not limited by the embodiments of the present invention.
For example, in the embodiment of the present invention, the memory size, the network condition, and the election parameters corresponding to the CPU performance of the storage servers (i.e., the storage server 1 in which the block 1 is located, the storage server 2 in which the block 2 is located, and the storage server 3 in which the block 3 is located) are obtained, where the election parameters may specifically be scores, which mean that the scores are different for different memory sizes, and similarly for the network condition, the CPU performance, and the like, the scores are different when the network condition is different, and the CPU performance is different.
Based on the above, the sum of the election parameters is obtained for the election parameters corresponding to the memory size, the network condition and the CPU performance of the storage server 1, or the weighted sum of the election parameters is obtained; acquiring the sum of election parameters or the weighted sum of the election parameters corresponding to the memory size, the network condition and the CPU performance of the storage server 2; acquiring the sum of election parameters or the weighted sum of the election parameters corresponding to the memory size, the network condition and the CPU performance of the storage server 3;
For the storage server 1 there is a corresponding sum of election parameters or a weighted sum of election parameters, for the storage server 2 there is a corresponding sum of election parameters or a weighted sum of election parameters, for the storage server 3 there is a corresponding sum of election parameters or a weighted sum of election parameters, and the target block is selected from the blocks of the target block group according to the sum of election parameters or the weighted sum of election parameters. And selecting a maximum value from the sum of the election parameters or the weighted sum of the election parameters, and then selecting a block corresponding to the maximum value from the blocks of the target block group as a target block.
Corresponding to the above method embodiment, the embodiment of the present invention further provides a data processing apparatus, as shown in fig. 5, where the apparatus may include: a data parsing module 510, a consensus group searching module 520, a data processing module 530.
The data parsing module 510 is configured to parse the original data carried in the data modification instruction when receiving the data modification instruction sent by the client;
the consensus group searching module 520 is configured to search a target consensus group from a preset consensus group set, where the target master block and the target slave block in the target consensus group both locally store the original data;
The method comprises the steps of selecting a master block and a slave block from a plurality of preset block groups respectively through a preset consensus algorithm to obtain a plurality of consensus groups, wherein the consensus groups form a consensus group set;
and the data processing module 530 is configured to send the data modification instruction to the target master block, so that the target master block executes the data modification instruction to modify the original data, and send a data modification instruction to the target slave block, so that the target slave block executes the data modification instruction to modify the original data.
The embodiment of the invention also provides an electronic device, as shown in fig. 6, which comprises a processor 61, a communication interface 62, a memory 63 and a communication bus 64, wherein the processor 61, the communication interface 62 and the memory 63 complete communication with each other through the communication bus 64,
a memory 63 for storing a computer program;
the processor 61 is configured to execute the program stored in the memory 63, and implement the following steps:
under the condition that a data modification instruction sent by a client is received, analyzing original data carried in the data modification instruction; searching a target consensus group from a preset consensus group set, wherein the original data are locally stored in a target main block and a target slave block in the target consensus group; the method comprises the steps of selecting a master block and a slave block from a plurality of preset block groups respectively through a preset consensus algorithm to obtain a plurality of consensus groups, wherein the consensus groups form a consensus group set; and sending the data modification instruction to the target main block so that the target main block executes the data modification instruction to modify the original data, and forwarding the data modification instruction to the target slave block so that the target slave block executes the data modification instruction to modify the original data.
The communication bus mentioned by the above electronic device may be a peripheral component interconnect standard (Peripheral Component Interconnect, abbreviated as PCI) bus or an extended industry standard architecture (Extended Industry Standard Architecture, abbreviated as EISA) bus, or the like. The communication bus may be classified as an address bus, a data bus, a control bus, or the like. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus.
The communication interface is used for communication between the electronic device and other devices.
The memory may include random access memory (Random Access Memory, RAM) or non-volatile memory (non-volatile memory), such as at least one disk memory. Optionally, the memory may also be at least one memory device located remotely from the aforementioned processor.
The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (Network Processor, NP for short), etc.; but also digital signal processors (Digital Signal Processing, DSP for short), application specific integrated circuits (Application Specific Integrated Circuit, ASIC for short), field-programmable gate arrays (Field-Programmable Gate Array, FPGA for short) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
In yet another embodiment of the present invention, a storage medium is provided, in which instructions are stored, which when run on a computer, cause the computer to perform the data processing method according to any of the above embodiments.
In a further embodiment of the present invention, a computer program product comprising instructions which, when run on a computer, causes the computer to perform the data processing method according to any of the above embodiments is also provided.
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present invention, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a storage medium or transmitted from one storage medium to another, for example, from one website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The storage media may be any available media that can be accessed by a computer or a data storage device such as a server, data center, or the like that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), etc.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.
The foregoing description is only of the preferred embodiments of the present invention and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims (12)

1. A method of data processing, the method comprising:
under the condition that a data modification instruction sent by a client is received, analyzing original data carried in the data modification instruction;
searching a target consensus group from a preset consensus group set, wherein the original data are locally stored in a target main block and a target slave block in the target consensus group;
the method comprises the steps of selecting a master block and a slave block from a plurality of preset block groups respectively through a preset consensus algorithm to obtain a plurality of consensus groups, wherein the consensus groups form a consensus group set;
and sending the data modification instruction to the target main block so that the target main block executes the data modification instruction to modify the original data, and forwarding the data modification instruction to the target slave block so that the target slave block executes the data modification instruction to modify the original data.
2. The method of claim 1, wherein the searching for the target consensus group from the preset consensus group set comprises:
Analyzing target data carried in the data modification instruction, wherein the target data comprises data modified for the original data;
acquiring a data modification quantity threshold value set for a data modification scene, and judging whether the data quantity of the target data is smaller than or equal to the data modification quantity threshold value;
if the data quantity of the target data is smaller than or equal to the data modification quantity threshold value, searching a target consensus group from a preset consensus group set.
3. The method of claim 1, wherein the searching for the target consensus group from the preset consensus group set comprises:
analyzing target data carried in the data modification instruction, wherein the target data comprises data modified for the original data;
acquiring a data grade set for a data modification scene, and judging whether the grade of the target data is greater than or equal to the data grade;
if the grade of the target data is greater than or equal to the grade of the data, searching a target consensus group from a preset consensus group set.
4. A method according to any one of claims 1 to 3, further comprising:
Receiving data modification success messages returned by the target main block and the target slave block, and counting the number of the data modification success messages;
judging whether the number of the data modification success messages is larger than or equal to a preset message number threshold value, wherein the preset message number threshold value is smaller than or equal to the number of elements in the target consensus group;
if the number of the data modification success messages is larger than or equal to the preset message number threshold, returning a message that the data modification instruction is successfully executed to the client;
and if the number of the data modification success messages is smaller than the preset message number threshold, returning a message of failure in executing the data modification instruction to the client.
5. The method according to claim 2, wherein the method further comprises:
if the data quantity of the target data is larger than the data modification quantity threshold value, searching a target block group from a plurality of block groups, wherein the blocks in the target block group locally store the original data;
selecting a target block from blocks included in the target block group, and reading the original data from the target block;
and modifying the original data according to the target data, and writing the modified original data into the blocks included in the target block group.
6. A method according to claim 3, characterized in that the method further comprises:
if the grade of the target data is smaller than the grade of the data, searching a target block group from a plurality of block groups, wherein the blocks in the target block group all store the original data;
selecting a target block from blocks included in the target block group, and reading the original data from the target block;
and modifying the original data according to the target data, and writing the modified original data into the blocks included in the target block group.
7. The method according to claim 5 or 6, wherein selecting the target block from the blocks included in the target block group includes:
acquiring a block selection threshold set for a data modification scene, and judging whether the data amount of the target data is smaller than or equal to the block selection threshold;
and if the data quantity of the target data is smaller than or equal to the block selection threshold value, randomly selecting a target block from the blocks included in the target block group according to a preset random selection rule.
8. The method of claim 7, wherein selecting the target block from the blocks included in the target block group further comprises:
If the data volume of the target data is larger than the block selection threshold, determining a storage server where each block in the target block group is located, and acquiring performance parameters of the storage server;
and randomly selecting a target block from the blocks of the target block group according to the performance parameters of the storage server.
9. The method of claim 8, wherein the performance parameter comprises at least one of: memory size, network conditions, CPU performance;
the selecting, according to the performance parameter of the storage server, a target block randomly from the blocks in the target block group includes:
acquiring election parameters corresponding to the memory size, the network condition and the CPU performance of the storage server;
and acquiring the sum of the election parameters, or acquiring a weighted sum of the election parameters, and selecting a target block from the blocks of the target block group according to the sum or the weighted sum.
10. A data processing apparatus, the apparatus comprising:
the data analysis module is used for analyzing the original data carried in the data modification instruction under the condition that the data modification instruction sent by the client is received;
The consensus group searching module is used for searching a target consensus group from a preset consensus group set, wherein the target main block and the target slave block in the target consensus group are locally stored with the original data;
the method comprises the steps of selecting a master block and a slave block from a plurality of preset block groups respectively through a preset consensus algorithm to obtain a plurality of consensus groups, wherein the consensus groups form a consensus group set;
and the data processing module is used for sending the data modification instruction to the target main block so that the target main block executes the data modification instruction to modify the original data, and forwarding the data modification instruction to the target slave block so that the target slave block executes the data modification instruction to modify the original data.
11. The electronic equipment is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are communicated with each other through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of any one of claims 1-9 when executing a program stored on a memory.
12. A storage medium having stored thereon a computer program, which when executed by a processor performs the method of any of claims 1-9.
CN202210039129.8A 2022-01-13 2022-01-13 Data processing method, device, electronic equipment and storage medium Pending CN116483843A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210039129.8A CN116483843A (en) 2022-01-13 2022-01-13 Data processing method, device, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210039129.8A CN116483843A (en) 2022-01-13 2022-01-13 Data processing method, device, electronic equipment and storage medium

Publications (1)

Publication Number Publication Date
CN116483843A true CN116483843A (en) 2023-07-25

Family

ID=87218257

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210039129.8A Pending CN116483843A (en) 2022-01-13 2022-01-13 Data processing method, device, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN116483843A (en)

Similar Documents

Publication Publication Date Title
CN108846749B (en) Partitioned transaction execution system and method based on block chain technology
CN107315825B (en) Index updating system, method and device
CN110659151B (en) Data verification method and device and storage medium
US20170185454A1 (en) Method and Electronic Device for Determining Resource Consumption of Task
CN110581887B (en) Data processing method, device, block chain node and storage medium
CN111046310A (en) Page processing method, device, server and computer readable storage medium
CN110677684B (en) Video processing method, video access method, distributed storage method and distributed video access system
US20200036812A1 (en) Resource Download Method, Electronic Device, and Apparatus
US20230030856A1 (en) Distributed table storage processing method, device and system
US9201609B2 (en) Efficient replication of changes to a byte-addressable persistent memory over a network
CN110910249B (en) Data processing method and device, node equipment and storage medium
CN113794764A (en) Request processing method and medium for server cluster and electronic device
CN111385255B (en) Asynchronous call implementation method and device, server and server cluster
CN111562884A (en) Data storage method and device and electronic equipment
WO2020029588A1 (en) Data reading method, device, system, and distributed system
CN112148206A (en) Data reading and writing method and device, electronic equipment and medium
CN110427394B (en) Data operation method and device
CN115576872B (en) Access detection method and device for multi-level cache
CN110765125B (en) Method and device for storing data
CN116483843A (en) Data processing method, device, electronic equipment and storage medium
CN111309693A (en) Data synchronization method, device and system, electronic equipment and storage medium
CN115811483A (en) Network state monitoring method and device, electronic equipment and storage medium
CN111400327B (en) Data synchronization method and device, electronic equipment and storage medium
CN110968267B (en) Data management method, device, server and system
CN114116656A (en) Data processing method and related device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination