CN115167769A

CN115167769A - Method, device and system for writing data

Info

Publication number: CN115167769A
Application number: CN202210779387.XA
Authority: CN
Inventors: 张俊峰; 张永锋; 郑静; 王佳
Original assignee: Jingdong Technology Information Technology Co Ltd
Current assignee: Jingdong Technology Information Technology Co Ltd
Priority date: 2022-07-04
Filing date: 2022-07-04
Publication date: 2022-10-11

Abstract

The invention discloses a method, a device and a system for writing data, and relates to the technical field of computers. One embodiment of the method comprises: acquiring a data writing request, wherein the data writing request indicates a target file to be written, a target position to be written and data to be written; generating an operation log corresponding to the data writing request according to the request identifier of the data writing request, the identifier of the target file, the target position and the writing state; and writing the data to be written into the target position in the target file, and updating the writing state in the operation log. The implementation method can avoid the write-in amplification of the data, improve the utilization rate of the disk bandwidth and ensure the consistency of the data.

Description

Method, device and system for writing data

Technical Field

The present invention relates to the field of computer technologies, and in particular, to a method, an apparatus, and a system for writing data.

Background

In the field of distributed storage, a copy of data is typically stored on multiple physical machines at the same time for high availability of the data.

The process of writing data to a physical machine is generally divided into two phases: the first stage is to write data into a log file; and in the second stage, the data is really in the file on the physical machine according to the user data and the metadata in the log file.

In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art:

in the existing writing process, data writing is carried out in two stages of writing a log file and actually writing a file, which causes the writing amplification of data and reduces the utilization rate of disk bandwidth.

Disclosure of Invention

In view of this, embodiments of the present invention provide a method, an apparatus, and a system for writing data, which can generate a corresponding operation log according to a request identifier, an identifier of a target file to be written, a target position to be written, and a writing status during a process of writing data, without writing the data to be written into a log file. Therefore, the data to be written can be written into the physical machines of the distributed cluster only by one-time writing operation, so that the writing amplification of the data is avoided, and the utilization rate of the disk bandwidth is improved; and moreover, the consistency of data written in each node can be ensured according to the operation logs on each node in the distributed cluster.

To achieve the above object, according to an aspect of an embodiment of the present invention, there is provided a method of writing data.

The method for writing data in the embodiment of the invention can be applied to any node of a distributed cluster; the method comprises the following steps: acquiring a data writing request, wherein the data writing request indicates a target file to be written, a target position to be written and data to be written;

generating an operation log corresponding to the data writing request according to the request identifier of the data writing request, the identifier of the target file, the target position and the writing state;

and writing the data to be written into the target position in the target file, and updating the writing state in the operation log.

Optionally, the generating an operation log corresponding to the data write request and the updating the write state in the operation log include:

generating a first operation log according to the writing state before the data to be written is written, the request identifier, the identifier of the target file and the target position;

and generating a second operation log according to the updated writing state, the request identifier, the identifier of the target file and the target position.

Optionally, a data write request sent by a user is received by a master node of the distributed cluster, and the data write request is sent to one or more slave nodes of the distributed cluster.

Optionally, the method further comprises:

receiving, by the master node, one or more operation logs respectively sent by the one or more slave nodes, where the operation logs include the second operation log or a combination of the first operation log and the second operation log;

comparing the received one or more operation logs with one or more operation logs stored by the user respectively;

and under the condition that the comparison result is inconsistent, updating the data written in the slave node according to the inconsistent target operation log.

Optionally, the updating the data written from the node according to the inconsistent target operation log includes:

reading target data on the main node according to the identification and the target position of the target file included in the target operation log;

and sending the target data to the slave node through the master node, so that the slave node updates the data in the target file stored by the slave node according to the target data.

Optionally, after the slave node generates the second operation log, the method further includes:

sending a response message of the data writing request to a master node through the slave node;

and the master node sends a write response message to the user under the condition that the number of the received response messages is determined to be not less than the number of the slave nodes.

To achieve the above object, according to still another aspect of embodiments of the present invention, there is provided an apparatus for writing data.

The device for writing data in the embodiment of the invention can be applied to any node of a distributed cluster, and comprises the following steps:

the device comprises a request acquisition module, a data writing module and a data writing module, wherein the request acquisition module is used for acquiring a data writing request which indicates a target file to be written, a target position to be written and data to be written;

a log generating module, configured to generate an operation log corresponding to the data write request according to the request identifier of the data write request, the identifier of the target file, the target location, and the write status;

and the data writing module is used for writing the data to be written into the target position in the target file and updating the writing state in the operation log.

To achieve the above object, according to still another aspect of embodiments of the present invention, there is provided a system for writing data.

The system for writing data in the embodiment of the invention comprises a main node and one or more slave nodes, wherein the main node and the slave nodes are provided with the data writing device provided by the second aspect;

the master node is used for receiving the operation logs sent by the one or more slave nodes; comparing the received one or more operation logs with one or more operation logs stored by the user respectively; and under the condition that the comparison result is inconsistent, updating the data written in the slave node according to the inconsistent target operation log.

To achieve the above object, according to still another aspect of an embodiment of the present invention, there is provided a server to write data.

The server for writing data in the embodiment of the invention comprises: one or more processors; a storage device for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement a method of writing data according to an embodiment of the present invention.

To achieve the above object, according to still another aspect of embodiments of the present invention, there is provided a computer-readable storage medium.

A computer-readable storage medium of an embodiment of the present invention has stored thereon a computer program that, when executed by a processor, implements a method of writing data of an embodiment of the present invention.

One embodiment of the above invention has the following advantages or benefits: and in the process of writing data, generating a corresponding operation log according to the request identifier, the identifier of the target file to be written, the target position to be written and the writing state, without writing the data to be written into the log file. Therefore, the data to be written can be written into the physical machines of the distributed cluster only by one-time writing operation, so that the writing amplification of the data is avoided, and the utilization rate of the disk bandwidth is improved; and moreover, the consistency of data written in each node can be ensured according to the operation logs on each node in the distributed cluster.

Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.

Drawings

The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:

FIG. 1 is a diagram illustrating the main steps of a method for writing data according to an embodiment of the present invention;

FIG. 2 is a diagram illustrating the main steps of another method for writing data according to an embodiment of the present invention;

FIG. 3 is a diagram illustrating the main steps of another method for writing data according to the embodiment of the present invention;

FIG. 4 is a schematic diagram of the main blocks of an apparatus for writing data according to an embodiment of the present invention;

FIG. 5 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;

fig. 6 is a schematic block diagram of a computer system suitable for use in implementing a terminal device or server of an embodiment of the invention.

Detailed Description

Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

It should be noted that embodiments of the present invention and technical features in the embodiments may be combined with each other without conflict.

Fig. 1 is a schematic diagram illustrating main steps of a method for writing data according to an embodiment of the present invention.

As shown in fig. 1, a method for writing data according to an embodiment of the present invention may be applied to any node of a distributed cluster, and mainly includes the following steps:

step S101: and acquiring a data writing request, wherein the data writing request indicates a target file to be written, a target position to be written and data to be written.

When data updating or data inserting operations are required, a user may initiate a data write request to a master node in the distributed cluster, where the data write request may include a destination file (file ID) to be written and a destination location to be written (e.g., a write range size, an offset, etc. in the file).

Step S102: and generating an operation log corresponding to the data writing request according to the request identifier of the data writing request, the identifier of the target file, the target position and the writing state.

After receiving a data write request, a master node in the distributed cluster may generate a corresponding request identifier (e.g., a sequence ID) for the data write request, where the sequence ID is an autonomous and unique number and may be used to uniquely identify the data write request. Meanwhile, the master node may also send the data write request to one or more slave nodes in the distributed cluster, and generally, in order to ensure high availability of data, the master node sends the data write request to multiple slave nodes, so that each of the multiple slave nodes performs a corresponding data write operation. Then, the slave node may also generate a request identifier of the data write request accordingly. It will be appreciated that since the request identifications of the master node and the slave node are self-increasing and unique, the request identifications generated by the slave node and the master node for the same data write request are the same. In addition, the master node may generate a request identifier of the data write request, and then transmit the data write request and the request identifier to the slave node.

Then, each of the master node and the slave node may generate an operation log corresponding to the data write request according to the request identifier (sequence ID) of the data write request, the identifier (file ID) of the destination file, the destination location to be written (including the write range size in the destination file, i.e., writing from one location to another location in the destination file, and the offset) and the write status (for example, start when data is not started), where the content of the operation log may be (sequence ID, file ID, size, offset, start).

Step S103: and writing the data to be written into the target position in the target file, and updating the writing state in the operation log.

After generating the oplog (sequence ID, file ID, size, offset, start), the master node and the slave node may start submitting data to actually write the data to the destination location in the destination file, and after the writing is completed, the write status in the oplog may be updated to update the oplog or generate a new oplog. For example, after the write is completed, if the write status is finish, the original operation log may be updated from (sequence ID, file ID, size, offset, start) to (sequence ID, file ID, size, offset, finish), or a new operation log may be directly generated as (sequence ID, file ID, size, offset, finish), where the write status finish identifies the completion of the write.

In a preferred embodiment of the present invention, the log is recorded by generating operation logs before and after writing, respectively, so as to ensure the integrity of the log. That is, before data writing, a first operation log (sequence ID, file ID, size, offset, start) is generated based on a write status before data to be written is written, a request identification, an identification of a destination file, and a destination position; after the data writing is completed, a second operation log (sequence id, file id, size, offset, finish) is generated according to the updated writing status (finish), the request identifier, the identifier of the destination file, and the destination location.

In addition, in an embodiment of the present invention, in order to ensure data consistency between the master node and the slave nodes, after completing the data writing operation and generating the operation logs, each slave node transmits the operation log generated by itself to the master node, where the operation log may be a second operation log or a combination of the first operation log and the second operation log, that is, the slave node may transmit only the second operation log (sequence id, fileID, size, offset, finish) to the master node, or may transmit both the first operation log (sequence id, fileID, size, offset, start) and the second operation log (sequence id, fileID, size, offset, finish). After receiving the operation logs sent by one or more slave nodes, the master node may compare the received one or more operation logs with one or more operation logs stored in the master node, respectively, and update the data written in the slave nodes according to inconsistent target operation logs in the case that the comparison results are inconsistent. Or the master node may compare the similarities and differences of the operation logs on the master node and the slave node by actively reading the operation logs on the slave node.

For example, the oplogs stored on the master node are: [1000, filter A,4096,0, start ], [1000, filter A,4096,0, finish ], [1001, filter A,4096,8192, start ], [1001, filter A,4096,8192, finish ]. The operation log sent from node a is: [1000, fileA,4096,0, start ]. By comparing the operation logs, the following can be found: slave node a lacks 2 writes and its data is inconsistent with the master node. And, it can be determined by comparing the operation logs that the 2 writes missing from the node a are the 2 writes corresponding to the request identifications sequence ids of 1000 and 1001.

After the operation logs are found to be inconsistent, the data written from the nodes can be updated according to the inconsistent target operation logs. Specifically, in an embodiment of the present invention, according to an identifier and a target position of a target file included in a target operation log, target data on a master node is read, and the target data is sent to a slave node corresponding to the target operation log by the master node, so that the slave node updates data in the target file stored by the slave node according to the target data.

For example, after it is determined that the oplogs are inconsistent, the slave node a may initiate a read request to the master node to obtain target data of the sequence ID 1000 and the sequence ID 1001, and then the slave node may perform data writing again according to the obtained target data, thereby ensuring that the data of the slave node is consistent with that of the master node.

It should be noted that, at any time point or when any node fails, the operation logs are compared to determine whether the data on the nodes in the distributed cluster are consistent. For example, when a certain slave node is abnormally halted and restarted, because the slave node may be halted while incomplete data writing is performed, after the slave node is restarted, the master node may check the progress of data writing performed by the slave node by comparing the operation logs, and enable the slave node to continue to perform data writing according to the progress before halt. For another example, in the process of processing data writing, the master node and the slave nodes are down and restarted, in this case, the master node is reselected, and after the master node is reselected, the master node can also determine the data writing progress of the slave nodes by comparing the operation logs, and the data on each slave node is kept consistent according to the data written by the master node. That is, after all nodes are down and restarted, the data on each slave node can be kept consistent according to the operation logs on the newly elected master nodes.

In addition, after each slave node completes data writing and generates a second operation log, a response message of a data writing request can be sent to the master node, the master node determines that each slave node completes data writing under the condition that the number of the received response messages is not smaller than the number of the slave nodes, and then sends a writing response message to the user to inform the user that the data writing is completed, so that the user is informed of the writing result after the data is written successfully, the user is prevented from being informed when the data writing is failed, and the user experience is improved.

It is to be understood that the slave node may transmit a response message to the master node along with the second oplog; the second operation log can also be used as a response message to be sent to the main node; or only sending a response message to the master node, after receiving the response message, the master node may determine that the slave node has completed data writing according to the response message, or actively read the operation log of the corresponding slave node according to the response message, and if the operation log on the slave node is the same as the operation log on the master node, the master node determines that the slave node has completed data writing again.

The data writing method provided by the embodiment of the present invention is described in detail below with an example in which a master node and two slave nodes (slave node a and slave node B) in a distributed cluster execute data writing operations corresponding to the same data writing request. As shown in fig. 2, a method for writing data according to an embodiment of the present invention may include the following steps:

step S201: the main node receives a data writing request sent by a user.

Step S202: the master node generates a request identifier of the data write request, and sends the request identifier and the data write request to the slave node a and the slave node B respectively.

For example, the request is identified as N.

Step S203: and the master node, the slave node A and the slave node B respectively generate a first operation log according to the request identifier, the identifier of the target file to be written, the target position to be written and the writing state.

The first operation log is an operation log before data is actually written, and may be recorded as Start Write N: (N, file ID, size, offset, start).

Step S204: and respectively writing the data to be written into a target position in the target file by the main node, the slave node A and the slave node B, and generating a second operation log.

Wherein the second operation log may be recorded as: finish Write N: (N, file ID, size, offset, finish).

Step S205: the slave node a and the slave node B send response messages to the master node.

In this embodiment, the slave node sends the response message to the master node only after the write is successful, and if the slave node goes down due to a failure in the write process, that is, if the data is not written completely, the slave node does not send the response message to the master node.

Step S206: and the master node determines that the number of the received response messages is not less than the number of the slave nodes, thereby determining that the data writing is successful, and sending a writing response message to the user.

In another embodiment of the present invention, the master node and/or the slave node may be down and restarted in the process of writing data, that is, the master node and/or the slave node is restarted when data writing is not completed, and at this time, the master node may perform data update on the operation log through comparison, so as to ensure data consistency of the master node and the slave node. The data writing method provided by the embodiment of the present invention is further described below by taking the case where the slave node a does not complete data writing as an example. As shown in fig. 3, the data writing method provided in the embodiment of the present invention includes the following steps:

step S301: the main node receives a data writing request sent by a user.

Step S302: the master node generates a request identifier of the data write request, and sends the request identifier and the data write request to the slave node a and the slave node B respectively.

For example, the request is identified as N.

Step S303: and the master node, the slave node A and the slave node B respectively generate a first operation log according to the request identifier, the identifier of the target file to be written, the target position to be written and the writing state.

Step S304: and respectively writing the data to be written into the target position in the target file by the master node and the slave node B, and generating a second operation log.

Wherein, the second operation log may be recorded as: finish Write N: (N, file ID, size, offset, finish).

Step S305: the slave node B sends a reply message to the master node.

In this example, since the slave node a does not complete the data writing, the second operation log cannot be generated, and the response message cannot be transmitted to the master node.

Step S306: and the master node judges that the number of the received response messages is less than that of the slave nodes.

It can be understood that, if the master node does not receive the response messages sent by all the slave nodes within the preset time period, the operation logs can be compared.

Step S307: and the master node determines the slave node A which does not send the response message, compares the operation log of the master node A with the operation log of the slave node A, and determines inconsistent target operation logs according to the comparison result.

For example, the oplogs stored on the master node are: [1000, file A,4096,0, start ], [1000, file A,4096,0, finish ], [1001, file A,4096,8192, start ], [1001, file A,4096,8192, finish ]. The oplog on slave node a is: [1000, fileA,4096,0, start ]. By comparing the operation logs, the following can be found: the slave node a lacks two writes and its data is inconsistent with the master node. And, it can be determined by comparing the operation logs that the two writes missing from the node a are the two writes corresponding to the request identifications N of 1000 and 1001.

Step S308: and reading target data on the main node according to the identification and the target position of the target file included in the target operation log.

Step S309: and the master node sends the target data to the slave node A, so that the slave node A updates the data in the target file according to the target data.

For example, after determining that the operation logs are inconsistent, the slave node a may initiate a read request to the master node to obtain the target data of N1000 and N1001, and then the slave node may perform data writing again according to the obtained target data, thereby ensuring that the data is consistent with that on the master node. It can be understood that, after the slave node a updates the data in the target file, the data writing is completed, at this time, the slave node a may send a response message to the master node again, and then the master node may send a write response message to the user after determining that the response messages of all the slave nodes are received.

According to the method for writing data in the embodiment of the invention, in the process of writing data, the corresponding operation log is generated according to the request identifier, the identifier of the target file to be written, the target position to be written and the writing state, and the data to be written is not written into the log file. Therefore, the data to be written can be written into the physical machines of the distributed cluster only by one-time writing operation, so that the writing amplification of the data is avoided, and the utilization rate of the disk bandwidth is improved; and moreover, the consistency of data written in each node can be ensured according to the operation logs on each node in the distributed cluster.

Fig. 4 is a schematic diagram of main blocks of an apparatus for writing data according to an embodiment of the present invention.

As shown in fig. 4, an apparatus 400 for writing data according to an embodiment of the present invention includes:

a request obtaining module 401, configured to obtain a data write request, where the data write request indicates a target file to be written, a target location to be written, and data to be written;

a log generating module 402, configured to generate an operation log corresponding to the data write request according to the request identifier of the data write request, the identifier of the target file, the target location, and the write status;

a data writing module 403, configured to write the data to be written into the target location in the target file, and update a writing status in the operation log.

In an embodiment of the present invention, the log generating module 402 is configured to generate a first operation log according to a writing status before the data to be written is written, the request identifier, the identifier of the target file, and the target location; and generating a second operation log according to the updated writing state, the request identifier, the identifier of the target file and the target position.

According to another embodiment of the present invention, there is also provided a system for writing data, including a master node and one or more slave nodes, where the master node and the slave nodes are provided with data writing devices provided in any of the above embodiments; wherein the content of the first and second substances,

In an embodiment of the present invention, a data write request sent by a user is received by a master node of the distributed cluster, and the data write request is sent to one or more slave nodes of the distributed cluster.

In an embodiment of the present invention, target data on the master node is read according to an identifier and a target location of a target file included in the target operation log; and sending the target data to the slave node through the master node, so that the slave node updates the data in the target file stored by the slave node according to the target data.

In one embodiment of the present invention, after the slave node generates the second operation log, a response message of the data write request is also sent to the master node through the slave node; and the master node sends a write response message to the user under the condition that the number of the received response messages is determined to be not less than the number of the slave nodes.

According to the device and the system for writing data, disclosed by the embodiment of the invention, in the process of writing data, the corresponding operation log is generated according to the request identifier, the identifier of the target file to be written, the target position to be written and the writing state, and the data to be written is not written into the log file. Therefore, the data to be written can be written into the physical machines of the distributed cluster only by one-time writing operation, so that the writing amplification of the data is avoided, and the utilization rate of the disk bandwidth is improved; and moreover, the consistency of data written in each node can be ensured according to the operation logs on each node in the distributed cluster.

Fig. 5 illustrates an exemplary system architecture 500 of a method of writing data or an apparatus for writing data to which embodiments of the present invention may be applied.

As shown in fig. 5, the system architecture 500 may include

terminal devices

501, 502, 503, a network 504, and a server 505. The network 504 is the medium used to provide communication links between

terminal devices

501, 502, 503 and the server 505. Network 504 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

The user may use the

terminal devices

501, 502, 503 to interact with a server 505 over a network 504 to receive or send messages or the like. The

terminal devices

501, 502, 503 may have various communication client applications installed thereon, such as a shopping application, a web browser application, a search application, an instant messaging tool, a mailbox client, social platform software, and the like.

The

terminal devices

501, 502, 503 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.

The server 505 may be a server providing various services, such as a background management server providing data support for requests sent by users using the

terminal devices

501, 502, 503. The backend management server may analyze the received request, and feed back a processing result (e.g., a data writing result) to the terminal device.

It should be noted that the method for writing data provided by the embodiment of the present invention is generally executed by the server 505, and accordingly, the apparatus for writing data is generally disposed in the server 505.

It should be understood that the number of terminal devices, networks, and servers in fig. 5 are merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

Referring now to FIG. 6, a block diagram of a computer system 600 suitable for use with a terminal device implementing embodiments of the present invention is shown. The terminal device shown in fig. 6 is only an example, and should not bring any limitation to the functions and the use range of the embodiment of the present invention.

As shown in fig. 6, the computer system 600 includes a Central Processing Unit (CPU) 601 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data necessary for the operation of the system 600 are also stored. The CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.

The following components are connected to the I/O interface 605: an input portion 606 including a keyboard, a mouse, and the like; an output portion 607 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. A driver 610 is also connected to the I/O interface 605 as needed. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 610 as necessary, so that a computer program read out therefrom is mounted in the storage section 608 as necessary.

In particular, according to embodiments of the present disclosure, the processes described above with reference to the flow diagrams may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer-readable medium, the computer program comprising program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 609 and/or installed from the removable medium 611. The computer program performs the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 601.

It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes a request acquisition module, a log generation module, and a data writing module. The names of these modules do not constitute a limitation to the modules themselves in some cases, and for example, the data writing module may also be described as a "module that writes data to be written to a target file".

As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not assembled into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: acquiring a data writing request, wherein the data writing request indicates a target file to be written, a target position to be written and data to be written; generating an operation log corresponding to the data writing request according to the request identifier of the data writing request, the identifier of the target file, the target position and the writing state; and writing the data to be written into the target position in the target file, and updating the writing state in the operation log.

According to the technical scheme of the embodiment of the invention, in the process of writing data, the corresponding operation log is generated according to the request identifier, the identifier of the target file to be written, the target position to be written and the writing state, and the data to be written is not written into the log file. Therefore, the data to be written can be written into the physical machines of the distributed cluster only by one-time writing operation, so that the writing amplification of the data is avoided, and the utilization rate of the disk bandwidth is improved; and moreover, the consistency of data written in each node can be ensured according to the operation logs on each node in the distributed cluster.

The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for writing data is applied to any node of a distributed cluster; the method comprises the following steps:

acquiring a data writing request, wherein the data writing request indicates a target file to be written, a target position to be written and data to be written;

2. The method according to claim 1, wherein the generating an operation log corresponding to the data write request and the updating a write status in the operation log comprise:

3. The method according to claim 1 or 2,

and receiving a data write-in request sent by a user through a master node of the distributed cluster, and sending the data write-in request to one or more slave nodes of the distributed cluster.

4. The method of claim 3, further comprising:

5. The method of claim 4, wherein updating the data written from the slave node according to the inconsistent target oplog comprises:

6. The method of claim 3, after the slave node generates the second oplog, further comprising:

and the master node sends a write response message to a user under the condition that the number of the received response messages is not less than the number of the slave nodes.

7. An apparatus for writing data, applied to any node of a distributed cluster, comprising:

the request acquisition module is used for acquiring a data writing request, wherein the data writing request indicates a target file to be written, a target position to be written and data to be written;

8. A system for writing data, comprising a master node and one or more slave nodes, wherein the master node and the slave nodes are provided with the data writing apparatus according to claim 7;

9. A server for writing data, comprising:

one or more processors;

a storage device for storing one or more programs,

the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method recited in any of claims 1-6.

10. A computer-readable medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-6.