CN105630808A - Distributed file system based file reading and writing method and node server - Google Patents
Distributed file system based file reading and writing method and node server Download PDFInfo
- Publication number
- CN105630808A CN105630808A CN201410602747.4A CN201410602747A CN105630808A CN 105630808 A CN105630808 A CN 105630808A CN 201410602747 A CN201410602747 A CN 201410602747A CN 105630808 A CN105630808 A CN 105630808A
- Authority
- CN
- China
- Prior art keywords
- data block
- verification
- target data
- data
- packet
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a distributed file system based file reading and writing method and a node server. The method comprises the steps of receiving a data access request sent by a client through a data node server, wherein the data access request comprises information of a target data block required to be read; obtaining the target data block matched with the information of the target data block, wherein the target data block comprises identifier information and first data information, and the identifier information comprises check attribute information; and sending the check attribute information and the first data information to the client. According to the method and the node server, the I/O frequency of a disk can be reduced, the data reading efficiency can be effectively improved, and the HDFS random reading performance can be improved.
Description
Technical field
The present invention relates to the technical field of a kind of file process, be specifically related to a kind of file reading based on distributed file system, a kind of file wiring method based on distributed file system and a kind of node server.
Background technology
Along with the fast development of information technology, the information of magnanimity needs, while reliable memory, to be also satisfied the demand quickly accessed by substantial amounts of user. Traditional storage scheme has increasingly been difficult in adapt to developing rapidly of business in recent years from framework, become bottleneck and the obstacle of business development, now HDFS (HadoopDistributedFileSystem, Hadoop distributed file system) arises at the historic moment.
HDFS is by an efficient distributed algorithm, by the access of data be stored distributed among a large amount of server, can also, by accessing on distribution each server in the cluster, be a subversive development of conventional store framework while reliably many backup storages.
The least unit that data are stored by HDFS is that the big file of its storage can be dispersed as a lot of block by block (block), HDFS. HDFS carries out in units of packet when carrying out block read-write, each packet is made up of several chunk ultimate unit of data check (chunk be by), each chunk is generated a verification and (checksum) and by verification with store (size of a chunk is 512byte in default situations, the verification of generation and be 4byte). With reference to the block storage mode shown in Fig. 1, each block is divided into data block checking file (meta file) of data block file (block file) and its correspondence, when reading the data in block file, first from the checksumheader of meta file, the information such as checking algorithm are determined, then read the content that block file is corresponding, finally read the verification in meta file and information.
It can be seen that the reading process of a block file from above-mentioned flow process, relate to 3 magnetic disc i/os (input and output). In random reading process, if disk operating needs about 10ms every time, 3 times is exactly 30ms, and performance is very low.
Summary of the invention
In view of the above problems, it is proposed that the present invention is to provide a kind of and overcome the problems referred to above or solve a kind of file reading based on distributed file system of the problems referred to above, a kind of file wiring method based on distributed file system and corresponding a kind of node server at least in part.
According to one aspect of the present invention, it is provided that a kind of file reading based on distributed file system, described method includes:
Back end server receives the data access request that client sends, and described data access request includes the information of the target data block that needs read;
The target data block of the information matches of acquisition and described target data block, described target data block includes identification information and the first data message, and described identification information includes verification attribute information;
Described verification attribute information and described first data message are sent to described client.
Alternatively, described first data message includes verification and the data checksum of fragment data chunk and correspondence, and wherein, described fragment data chunk and described verification and data checksum are stored in the same storage medium of disk.
Alternatively, the step that described verification attribute information and described first data message send extremely described client is included:
From the identification information of described target data block, read described verification attribute information send to described client;
From described storage medium, read the default fragment data chunk of size and the verification of correspondence and data checksum sends to described client.
Alternatively, the information of described target data block includes target data block ID, the timestamp GenerationStamp of target data block, and the step of the target data block of the information matches of described acquisition and described target data block includes:
According to the timestamp GenerationStamp of described target data block ID and described target data block, the data block of storage in traversal disk, described data block includes data block ID and data block timestamp;
Search described data block ID identical with described target data block ID, and the described data block timestamp data block identical with the timestamp GenerationStamp of described target data block is as target data block.
Alternatively, described verification attribute information includes verification and version Inlinechecksumversion, verification and type checksumtype and each verification and byte number bytesPerChecksum.
Alternatively, the information of described target data block includes the block bias internal amount startOffset of target data block and the length of target data block, described presetting is sized to the size of the first packet packet, and described reading from described storage medium presets the fragment data chunk of size and the verification of correspondence and step that data checksum sends to described client includes:
Sub-step S11, creates the first packet packet;
Sub-step S12, calculates original position according to the block bias internal amount startOffset of described target data block;
Sub-step S13, from described original position, described storage medium is successively read each fragment data chunk and with in described verification corresponding for fragment data chunk and data checksum to described first packet packet, until fill up described first packet packet;
Sub-step S14, sends described first packet packet to client;
Sub-step S15, repeats above-mentioned sub-step S11 to sub-step S14, until sending the size length equal to described target data block of the data to client.
Alternatively, after described verification and data checksum position in described storage medium are positioned at the storage position of the fragment data chunk of correspondence, before being positioned at the storage position of next fragment data chunk of the fragment data chunk of described correspondence.
Alternatively, before described acquisition with the step of the target data block of the information matches of described target data block, also include:
Set up according to described data access request and be connected with the socket of described client.
According to a further aspect in the invention, it is provided that a kind of file wiring method based on distributed file system, described method includes:
Back end server receives the second packet packet, described second packet packet of client transmission and includes target data block ID, verification attribute information and the second data message;
According to the target data block that described target data block ID location is corresponding;
Described target data block ID adds described verification attribute information, generates the identification information of described target data block;
Described second data message is write in described target data block.
Alternatively, described second data message includes multiple fragment data chunk, and respectively with the plurality of verification corresponding for fragment data chunk and data checksum, described by the step that described second data message writes in described target data block be:
Write in described target data block corresponding for target data block ID by described fragment data chunk and with described verification corresponding for fragment data chunk and data checksum, wherein, described fragment data chunk and being stored in the same storage medium of disk with described verification corresponding for fragment data chunk and data checksum.
Alternatively, described verification attribute information includes verification and version Inlinechecksumversion, verification and type checksumtype and each verification and byte number bytesPerChecksum.
Alternatively, described described second data message write the step in described target data block after, also include:
Described second packet packet is transferred in next the back end server in the node queue previously generated by current data node server.
Alternatively, described method also includes:
Generating response message, described response message is inform the information that the current second packet packet process of described client completes;
Described response message is returned described client.
According to a further aspect in the invention, it is provided that a kind of node server, described server includes:
Request receiver module, is suitable to receive the data access request that client sends, and described data access request includes the information of the target data block that needs read;
Locating module, is suitable to obtain the target data block of the information matches with described target data block, and described target data block includes identification information and the first data message, and described identification information includes verification attribute information;
Data transmission blocks, is suitable to send to described client described verification attribute information and described first data message.
Alternatively, described first data message includes verification and the data checksum of fragment data chunk and correspondence, and wherein, described fragment data chunk and described verification and data checksum are stored in the same storage medium of disk.
Alternatively, data transmission blocks includes:
Attribute information reading submodule, is suitable to read described verification attribute information from the identification information of described target data block and sends to described client;
Digital independent submodule, is suitable to the default fragment data chunk of size of reading from described storage medium and the verification of correspondence and data checksum sends to described client.
Alternatively, the information of described target data block includes target data block ID, the timestamp GenerationStamp of target data block, and described locating module is further adapted for:
According to the timestamp GenerationStamp of described target data block ID and described target data block, the data block of storage in traversal disk, described data block includes data block ID and data block timestamp;
Search described data block ID identical with described target data block ID, and the described data block timestamp data block identical with the timestamp GenerationStamp of described target data block is as target data block.
Alternatively, described verification attribute information includes verification and version Inlinechecksumversion, verification and type checksumtype and each verification and byte number bytesPerChecksum.
Alternatively, the information of described target data block includes the block bias internal amount startOffset of target data block and the length of target data block, the described default size being sized to the first packet packet, and described data read module is further adapted for:
Create the first packet packet;
Original position is calculated according to the block bias internal amount startOffset of described target data block;
From described original position, described storage medium is successively read each fragment data chunk and with in described verification corresponding for fragment data chunk and data checksum to described first packet packet, until fill up described first packet packet;
Described first packet packet is sent to client.
Alternatively, after described verification and data checksum position in described storage medium are positioned at the storage position of the fragment data chunk of correspondence, before being positioned at the storage position of next fragment data chunk of the fragment data chunk of described correspondence.
Alternatively, described node server also includes:
Connection establishment module, is suitable to set up according to described data access request be connected with the socket of described client.
According to a further aspect in the invention, it is provided that a kind of node server, described node server includes:
Data reception module, the second packet packet, described second packet packet that are suitable to receive client transmission include target data block ID, verification attribute information and the second data message;
Data block locating module, is suitable to the target data block corresponding according to described target data block ID location;
Identification information generation module, is suitable to add described verification attribute information in described target data block ID, generates the identification information of described target data block;
Writing module, is suitable to write in described target data block described second data message.
Alternatively, described second data message includes multiple fragment data chunk, and respectively with the plurality of verification corresponding for fragment data chunk and data checksum, said write module is further adapted for:
Write in described target data block corresponding for target data block ID by described fragment data chunk and with described verification corresponding for fragment data chunk and data checksum, wherein, described fragment data chunk and being stored in the same storage medium of disk with described verification corresponding for fragment data chunk and data checksum.
Alternatively, described verification attribute information includes verification and version Inlinechecksumversion, verification and type checksumtype and each verification and byte number bytesPerChecksum.
Alternatively, described node server also includes:
Data transfer module, is suitable to current data node server and is transferred in next the back end server in the node queue previously generated by described second packet packet.
Alternatively, described node server also includes:
Response message generation module, is suitable to generate response message, and described response message is inform the information that the current second packet packet process of described client completes;
Response message sending module, is suitable to described response message is returned described client.
In embodiments of the present invention, by adding verification and version number in data block identifier information, verification and type, the verification attribute information such as each verification and byte number, and, the reading performance of HDFS is optimized by fragment data chunk and verification are stored in a memory element (such as in a block file) with checksum, make to read in the process of Block at DataNode, verification attribute information can be obtained from data block identifier information, then read correspondence chunk data and verification and, thus decreasing I/O number of disk, it is effectively improved the efficiency of digital independent, improve the HDFS performance read at random, such as, data for 500KB, before optimizing, HDFS reads needs cost 32ms, cost 15ms is had only to after optimization, for the data of 100KB, before optimizing, HDFS reads needs cost 30ms, has only to cost 14ms after optimization, and reading performance is obviously improved.
Described above is only the general introduction of technical solution of the present invention, in order to better understand the technological means of the present invention, and can be practiced according to the content of description, and in order to above and other objects of the present invention, feature and advantage can be become apparent, below especially exemplified by the specific embodiment of the present invention.
Accompanying drawing explanation
By reading hereafter detailed description of the preferred embodiment, various other advantage and benefit those of ordinary skill in the art be will be clear from understanding. Accompanying drawing is only for illustrating the purpose of preferred implementation, and is not considered as limitation of the present invention. And in whole accompanying drawing, it is denoted by the same reference numerals identical parts. In the accompanying drawings:
Fig. 1 illustrates the block storage mode in background technology according to an embodiment of the invention;
Fig. 2 illustrates the flow chart of steps of a kind of according to an embodiment of the invention file reading embodiment based on distributed file system;
Fig. 3 illustrates the first data information memory mode schematic diagram of a kind of according to an embodiment of the invention file reading embodiment based on distributed file system;
Fig. 4 illustrates the flow chart of steps of a kind of according to an embodiment of the invention file wiring method embodiment based on distributed file system;
Fig. 5 illustrates the structured flowchart of a kind of according to an embodiment of the invention node server embodiment one;
Fig. 6 illustrates the structured flowchart of a kind of according to an embodiment of the invention node server embodiment two.
Detailed description of the invention
It is more fully described the exemplary embodiment of the disclosure below with reference to accompanying drawings. Although accompanying drawing showing the exemplary embodiment of the disclosure, it being understood, however, that may be realized in various forms the disclosure and should do not limited by embodiments set forth here. On the contrary, it is provided that these embodiments are able to be best understood from the disclosure, and complete for the scope of the present disclosure can be conveyed to those skilled in the art.
Distributed file system HDFS is the core sub-project of Hadoop project, is the basis of data storage management in Distributed Calculation, is based on flow data mode access and processes the demand of super large file and develop, it is possible to running on cheap commercial server. The height that it has is fault-tolerant, high reliability, enhanced scalability, high acquired, high-throughput etc. are characterized as that mass data provides the storage not being afraid of fault, processes for the application of super large data set (LargeDataSet) and brings a lot of facility.
The structure of HDFS is host-guest architecture, one HDFS cluster is (to be called for short namenode by a namenode server, or claim metadata node server (abbreviation metadata node), NameNode), it is a management file name space and the master server regulating client-access file, certainly some back end servers are also had (to be called for short back end, DataNode), usually one machine of a node, it manages the storage of corresponding node, back end timing sends heart beating to namenode, to maintain the communication between namenode.
HDFS opening file name space also allows user data to store with document form. The internal mechanism of HDFS is that a file division is become one or more pieces of block, and these blocks are stored in one group of back end. Namenode is used for file or the directory operation of operation file NameSpace, as opened, closes, and renaming etc., it determines the mapping of block and back end simultaneously. Back end is responsible for the read-write requests from file system client, and back end also to perform the establishment of block simultaneously, deletes, and the block duplicate instructions from namenode.
Datanode stores the data in the local disk structure of dataserver. System can have multiple storage catalogue, and its storage catalogue can save its catalogue placing data block block by the configuration item dfs.data.dir of configuration file conf/hdfs-site.xml file, separates with ", ", it is possible to arrange multinomial. Each item is called storage catalogue, and on Datanode, the space of its correspondence is a Volume, manages with FSVolume, and all of storage catalogue FSVolumeSet manages.
User can pass through client Client and send read-write requests, and client can pass through network after receiving the request of user, mutual with NameNode and DataNode, transmission read-write requests. Wherein, client Client is the HDFS lib storehouse provided, and the application program of user needs to comprise this storehouse, calls the function of this storehouse offer to access NameNode and DataNode.
With reference to Fig. 2, it is shown that the flow chart of steps of a kind of file reading embodiment based on distributed file system of one embodiment of the invention, specifically may include steps of:
Step 101, back end server receives the data access request that client sends, and described data access request includes the information of the target data block that needs read;
In implementing, before client communicates with back end DataNode, first communicating with namenode NameNode, client with the interaction of namenode can be: client initiates RPC request to long-range NameNode; NameNode can optionally return the some or all of block list of file, all can return to the DataNode address of this block copy for each block, NameNode.
Client is chosen from the immediate DataNode of client to read block after obtaining block list; If client is DataNode inherently, then data will be directly read from this locality. Concrete, client reads the data in DataNode by sending data access request to DataNode, and wherein, data access request can include the information of the target data block that needs read, it is possible to is expressed as instruction:
| version | operator | blockID | generationstamp | startoffset | length | clientname | accesstoken |
Above-mentioned each meaning of parameters is as follows:
Whether the data required for Version:Client are consistent with the version of Datanode data presented;
Operation required for Operator:client, reads a block, one block of write etc.;
BlockId: the blockID (target data block ID) of block will be read;
The timestamp (the timestamp GenerationStamp of target data block) of generationStamp: required reading block;
StartOffset: block bias internal amount (the block bias internal amount startOffset of target data block);
Length: read the length (length of target data block) of block;
Clientname: the name of client;
The checking information that accesstoken:Client provides, user name password etc.
And in DataNode, after DataNode receives data access request, be first depending on the foundation of described data access request and be connected with the socket of described client. Concrete, it is a socketserver that DataNode comprises DataXceiverServer, DataXceiverServer, is responsible for receiving the client socket initiated and connects. After DataXceiverServer receives a socket connection, start a thread DataXceiver, DataXceiver be specifically responsible for the process of this scoket. DataXceiver reads the client operation code wanting to be operated from client, if operation code is OP_READ_BLOCK, then DataXceiver is responsible for reading this block, and sends the data to client.
Further, after DataXceiver receives data access request, being successively read each variable according to the form of above-mentioned instruction, after having read, if needing to conduct interviews control, being then verified according to the Client accesstoken provided. After being verified, according to one BlockSender example of information creating of described target data block, carried out the transmission of data by blockSender.
Step 102, it is thus achieved that with the target data block of the information matches of described target data block;
After DataNode receives file access request, it is possible to according to file access request location target data block, in a kind of preferred embodiment of the embodiment of the present invention, step 102 can include following sub-step:
Sub-step S10, according to the timestamp GenerationStamp of described target data block ID and described target data block, the data block of storage in traversal disk, described data block includes data block ID and data block timestamp;
Sub-step S20, searches described data block ID identical with described target data block ID, and the described data block timestamp data block identical with the timestamp GenerationStamp of described target data block is as target data block.
Being applied to the embodiment of the present invention, the identification information (i.e. data block title) of data block includes verification attribute information. Specifically, the data block that DataNode stores in local disk, containing verification attribute information in its title (or identification information), the identification information of data block can be expressed as follows: blk_id_generationStamp_Inlinechecksumversion_checksumtyp e_bytesPerChecksum; Wherein, blk_id is data block ID, and it can be a string random digit; GenerationStamp is the timestamp of data block, Inlinechecksumversion is verification and version (or claiming inline verification and version), can value being 1 or 2,1 be expressed as non-inlinechecksum version, 2 be expressed as the inlinechecksum version improved; Checksumtype is verification and type, it is possible to value is 0,1 ,-1, and wherein, 0 represents do not have checksum, and 1 represents CRC check, and-1 represents unknown verification mode; BytesPerChecksum represents each verification and byte number.
Such as, the identification information of certain data block is: blk_6151475469424452664_2267176246_2_1_512, then block_id is 6151475469424452664, and the timestamp of data block is 2267176246, checksum type: 1, the byte number of each checksum verification: 512.
After DataNode receives file access request, the target data block ID carried according to file access request and the timestamp GenerationStamp of described target data block, the data block of storage in traversal local disk, the name of described data block is searched data block ID identical with target data block ID, and the data block that data block timestamp is identical with the timestamp GenerationStamp of target data block, if searching successfully, then this data block found is target data block.
Verification attribute information (is namely verified and version Inlinechecksumversion by the embodiment of the present invention, verification and type checksumtype, and each verification and byte number bytesPerChecksum) add in the identification information of data block, make identification information except providing the information of the mark data block such as data block ID and data block timestamp, also provide for the relevant information of verification sum, so when navigating to concrete target data block, verification and relevant information can be obtained from the identification information of this target data block, while saving memory space, improve data acquisition efficiency, improve the performance of distributed file system on the whole.
Additionally, in the embodiment of the present invention, target data block can also include the first data message, and the first data message can include the fragment data chunk ultimate unit of data check (chunk be by) further and (size of a chunk is 512byte in default situations with each verification corresponding for fragment data chunk and data checksum, generate verification and be 4byte), and fragment data chunk and verification and data checksum are stored in the same storage medium (such as a block file) of disk, fragment data chunk and storage verification and data checksum is stored respectively without separately two files, thus being conducive to effectively managing the first data message, and it is effectively improved the efficiency of digital independent.
In practice, as shown in the first data information memory mode schematic diagram of Fig. 3, after verification and data checksum position in storage medium are positioned at the storage position of the fragment data chunk of correspondence, before being positioned at the storage position of next fragment data chunk of the fragment data chunk of correspondence.
Step 103, sends described verification attribute information and described first data message to described client;
After navigating to concrete target data block, it is possible to obtain verification attribute information from the title of target data block, and verification attribute information and the first data message are sent to client. In a kind of preferred embodiment of the embodiment of the present invention, step 103 can include following sub-step:
Sub-step S11, reads described verification attribute information from the identification information of described target data block and sends to described client;
After orienting target data block, it is possible to obtain the verification attribute information of this target data block from the identification information of target data block, and by blockSender, this verification attribute information is returned client, the verification mode concrete to inform client.
Sub-step S12, reads the default fragment data chunk of size and the verification of correspondence from described storage medium and data checksum sends to described client.
In a kind of preferred embodiment of the embodiment of the present invention, default size can be the size of the first packet packet, and sub-step S12 may further include following sub-step:
Sub-step S121, creates the first packet packet;
Specifically, before starting to read the first data message, it is possible to create the first packet packet, for depositing the first data message of reading. One packet is the ultimate unit that data send, and a packet can be made up of multiple chunk.
The structure of the first packet packet can be expressed as follows:
| packetLen | offset | sequenceNum | isLastPacket | startOffset | dataLen | checksum | data |
Wherein, the implication of each parameter is as follows:
The length of packetLen:intpacket, including fragment data chunk, verification and checksum etc.
Offset:longpacket side-play amount in block
This packet of the sequenceNum:long sequence number when current block reads
Whether isLastPacket:bytepacket is last
The length of the comprised block data of this packet of dataLen:int, clear data does not include verifying and other
Checksum: the verification of this each chunk of packet and, have how many chunk just have how many verification and
Data: the block data that this packet comprises
Wherein, the mark of DTD is a packetLen length is the bag of 0.
Sub-step S122, calculates original position according to the block bias internal amount startOffset of described target data block;
In implementing, data access request carries the block bias internal amount startOffset of target data block, after finding target data block, for target data block, calculate original position offset (the revised skew according to chunk object) according to the block bias internal amount startOffset of target data block. It should be noted that, offset and startOffset is distinctive in that startOffset is the original data the to read side-play amounts at block, and offset is the side-play amount after aliging according to the size of chunk according to startOffset, thus navigating to chunk more accurately.
Sub-step S123, from described original position, described storage medium is successively read each fragment data chunk and with in described verification corresponding for fragment data chunk and data checksum to described first packet packet, until fill up described first packet packet;
Sub-step S124, sends described first packet packet to client;
Sub-step S125, repeats above-mentioned sub-step S121-sub-step S124, until sending the size length equal to described target data block of the data to client.
In the implementation, it is possible to from original position, it is successively read each fragment data according to the order of fragment data chunk-verification and data checksum, when reading each fragment data, reads verification and the data checksum of this fragment data.
In the process reading fragment data chunk and verification and data checksum, can according to the structure of above-mentioned first packet packet, one by one fragment data chunk and verification are filled in the first packet packet with data checksum, until the size of the data filled reaches the size of the first packet packet, then represent that this first packet packet is full, now can pass through blockSender and the first packet packet that this is full is sent to client.
If the first data message sent to client is less than the length of target data block, then create a first packet packet again, and all the other fragment datas chunk and verification are filled in the first packet packet with data checksum, until the size of the data filled reaches the size of the first packet packet, send this first packet packet, repeat above-mentioned action, until the first data message sent to client is equal to the length of target data block. It should be noted that the size of the data in last first packet packet is likely less than the size setting the first packet packet.
In embodiments of the present invention, by adding verification and version number in data block identifier information, verification and type, the verification attribute information such as each verification and byte number, and, the reading performance of HDFS is optimized by fragment data chunk and verification are stored in a memory element (such as in a block file) with checksum, make to read in the process of Block at DataNode, verification attribute information can be obtained from data block identifier information, then read correspondence chunk data and verification and, thus decreasing I/O number of disk, it is effectively improved the efficiency of digital independent, improve the HDFS performance read at random, such as, data for 500KB, before optimizing, HDFS reads needs cost 32ms, cost 15ms is had only to after optimization, for the data of 100KB, before optimizing, HDFS reads needs cost 30ms, has only to cost 14ms after optimization, and reading performance is obviously improved.
With reference to Fig. 4, it is shown that the flow chart of steps of a kind of file wiring method embodiment based on distributed file system of one embodiment of the invention, specifically may include steps of:
Step 201, back end server receives the second packet packet that client sends, and wherein, described second packet packet includes target data block ID, verification attribute information and the second data message;
In implementing, client sends fileinfo to before back end server DataNode, and client first carries out following workflow:
1. client uses the HDFS client development library Client provided to initiate RPC request to long-range Namenode, and request creates file; After 2.Namenode receives request, check whether the file to create exists, and whether founder has permission carries out associative operation, and when checking as successful result, Namenode is one record of document creation, otherwise can allow client throw exception; 3. client applies for new blocks to Namenode, obtains the suitable datanodes list being used for stored copies replicas, and the setting of number of copies replication is determined by the size of list according in Namenode.
After client obtains datanodes list, then can carry out the write operation of the second packet packet, its process can be: creates a second packet packet, after in buffer memory, the data of write reach a certain size (chunk size), calculate verification and the data checksum of these chunk data, the verification of this chunk and correspondence and data checksum are write in the second packet packet, when the second packet packet is full, this packet is added to (dataQueue stores packet to be sent) in data queue dataQueue, whether the DataStreamer thread timing inquiry dataQueue in client has packet, if having, then this packet is transferred to another waiting list ackQueue packet to be answered such as (this queue) comprising from dataQueue, so far, start to send the data of this packet: development library issues first DataNode the second packet packet, enclose the information of other DataNode simultaneously.
Now, first DataNode works as follows: communicate with NameNode, registers the information of oneself; Creating DataXceiverServer, DataXceiverServer and often accept a socket request, a newly created DataXceiver processes this request; Being applied to the embodiment of the present invention, back end server DataNode adopts DataXceiver to receive the second packet packet.
As a kind of preferred exemplary of the embodiment of the present invention, the second data message can include multiple fragment data chunk, and respectively with the plurality of verification corresponding for fragment data chunk and data checksum; Verification attribute information can include verification and version Inlinechecksumversion, verification and type checksumtype and each verification and byte number bytesPerChecksum. Wherein, verification and version Inlinechecksumversion can value be 1 or 2,1 be expressed as non-inlinechecksum version, and 2 are expressed as the inlinechecksum version improved; Checksumtype can value be 0,1 ,-1, and wherein, 0 represents do not have checksum, and 1 represents CRC check, and-1 represents unknown verification mode.
Step 202, according to the target data block that described target data block ID location is corresponding;
Back end server is after receiving the second packet packet, corresponding target data block can be searched according to the target data block ID comprised in the second packet packet, specifically, back end server can inquire about the data block ID identical with target data block ID in the local database, using this data block corresponding for identical data block ID as target data block.
Step 203, adds described verification attribute information in described target data block ID, generates the identification information of described target data block;
Verification attribute information (is namely verified and version Inlinechecksumversion by the embodiment of the present invention, verification and type checksumtype, and each verification and byte number bytesPerChecksum) add in the identification information of data block, make identification information except providing the information of the mark data block such as data block ID and data block timestamp, also provide for the relevant information of verification sum, so when navigating to concrete target data block, verification and relevant information can be obtained from the identification information of this target data block, while saving memory space, improve data acquisition efficiency, improve the performance of distributed file system on the whole.
Step 204, writes described second data message in described target data block.
After navigating to target data block, it is possible to write the second data message under this target data block. In a kind of preferred embodiment of the embodiment of the present invention, step 204 can be:
Write in described target data block corresponding for target data block ID by described fragment data chunk and with described verification corresponding for fragment data chunk and data checksum, wherein, described fragment data chunk and being stored in the same storage medium of disk with described verification corresponding for fragment data chunk and data checksum.
As shown in the first data information memory mode of Fig. 3, after verification and data checksum position in storage medium are positioned at the storage position of the fragment data chunk of correspondence, before being positioned at the storage position of next fragment data chunk of the fragment data chunk of correspondence.
Fragment data chunk and verification are stored in the same storage medium (such as a block file) of disk with data checksum by the embodiment of the present invention, fragment data chunk and storage verification and data checksum is stored respectively without separately two files, thus being conducive to effectively managing the first data message, and it is effectively improved the efficiency of digital independent.
In practice, after DataNode receives the second packet packet, described verification and data checksum can be verified, after being proved to be successful, just fragment data chunk and verification are stored in the same storage medium of disk with data checksum, otherwise, notice client throw exception.
In implementing, development library writes first datanode in a streaming manner the second packet packet, after this datanode stores this second packet packet, pass it to the next datanode in pipeline again, a to the last datanode, this mode writing data is the form of streamline.
After last datanode successfully stores the second packet packet, a response message ackpacket can be generated, client it is back in pipeline, development library internal maintenance in client " ackqueue ", can remove corresponding packet from " ackqueue " after having successfully received the datanode ackpacket returned.
If in transmitting procedure, certain datanode is had to occur in that fault, so current pipeline can be closed, the datanode broken down can remove from current pipeline, remaining block may proceed to continue in remaining datanode to transmit with the form of pipeline, Namenode can distribute a new datanode simultaneously, keeps the replicas quantity set.
For the embodiment of the method for Fig. 4, due to itself and the embodiment of the method basic simlarity of Fig. 2, so what describe is fairly simple, relevant part illustrates referring to the part of embodiment of the method.
For embodiment of the method, in order to be briefly described, therefore it is all expressed as a series of combination of actions, but those skilled in the art should know, the present invention is not by the restriction of described sequence of movement, because according to the present invention, some step can adopt other orders or carry out simultaneously. Secondly, those skilled in the art also should know, embodiment described in this description belongs to preferred embodiment, necessary to involved action and the module not necessarily present invention.
With reference to Fig. 5, it is shown that the structured flowchart of a kind of node server embodiment one of one embodiment of the invention, described node server can include such as lower module:
Request receiver module 501, is suitable to receive the data access request that client sends, and described data access request includes the information of the target data block that needs read;
Locating module 502, is suitable to obtain the target data block of the information matches with described target data block, and described target data block includes identification information and the first data message, and described identification information includes verification attribute information;
Data transmission blocks 503, is suitable to send to described client described verification attribute information and described first data message.
In a kind of preferred embodiment of the embodiment of the present invention, described first data message includes verification and the data checksum of fragment data chunk and correspondence, wherein, described fragment data chunk and described verification and data checksum are stored in the same storage medium of disk.
In a kind of preferred embodiment of the embodiment of the present invention, data transmission blocks 503 can include following submodule:
Attribute information reading submodule, is suitable to read described verification attribute information from the identification information of described target data block and sends to described client;
Digital independent submodule, is suitable to the default fragment data chunk of size of reading from described storage medium and the verification of correspondence and data checksum sends to described client.
In a kind of preferred embodiment of the embodiment of the present invention, the information of described target data block includes target data block ID, the timestamp GenerationStamp of target data block, and described locating module 502 is further adapted for:
According to the timestamp GenerationStamp of described target data block ID and described target data block, the data block of storage in traversal disk, described data block includes data block ID and data block timestamp;
Search described data block ID identical with described target data block ID, and the described data block timestamp data block identical with the timestamp GenerationStamp of described target data block is as target data block.
As a kind of preferred exemplary of the embodiment of the present invention, described verification attribute information can include verification and version Inlinechecksumversion, verification and type checksumtype and each verification and byte number bytesPerChecksum.
In a kind of preferred embodiment of the embodiment of the present invention, the information of described target data block includes the block bias internal amount startOffset of target data block and the length of target data block, the described default size being sized to the first packet packet, described data read module 504 is further adapted for:
Create the first packet packet;
Original position is calculated according to the block bias internal amount startOffset of described target data block;
From described original position, described storage medium is successively read each fragment data chunk and with in described verification corresponding for fragment data chunk and data checksum to described first packet packet, until fill up described first packet packet;
Described first packet packet is sent to client.
It is applied to the embodiment of the present invention, after described verification and data checksum position in described storage medium are positioned at the storage position of the fragment data chunk of correspondence, before being positioned at the storage position of next fragment data chunk of the fragment data chunk of described correspondence.
In a kind of preferred embodiment of the embodiment of the present invention, described node server also includes:
Connection establishment module, is suitable to set up according to described data access request be connected with the socket of described client.
For the server example of Fig. 5, due to itself and the embodiment of the method basic simlarity of Fig. 2, so what describe is fairly simple, relevant part illustrates referring to the part of embodiment of the method.
With reference to Fig. 6, it is shown that the structured flowchart of a kind of node server embodiment two of one embodiment of the invention, described node server can include such as lower module:
Data reception module 601, the second packet packet, described second packet packet that are suitable to receive client transmission include target data block ID, verification attribute information and the second data message;
Data block locating module 602, is suitable to the target data block corresponding according to described target data block ID location;
Identification information generation module 603, is suitable to add described verification attribute information in described target data block ID, generates the identification information of described target data block;
Writing module 604, is suitable to write in described target data block described second data message.
In a kind of preferred embodiment of the embodiment of the present invention, described second data message includes multiple fragment data chunk, and respectively with the plurality of verification corresponding for fragment data chunk and data checksum, said write module 604 is further adapted for:
Write in described target data block corresponding for target data block ID by described fragment data chunk and with described verification corresponding for fragment data chunk and data checksum, wherein, described fragment data chunk and being stored in the same storage medium of disk with described verification corresponding for fragment data chunk and data checksum.
As a kind of preferred exemplary of the embodiment of the present invention, described verification attribute information includes verification and version Inlinechecksumversion, verification and type checksumtype and each verification and byte number bytesPerChecksum.
In a kind of preferred embodiment of the embodiment of the present invention, described node server also includes:
Data transfer module, is suitable to current data node server and is transferred in next the back end server in the node queue previously generated by described second packet packet.
In a kind of preferred embodiment of the embodiment of the present invention, described node server also includes:
Response message generation module, is suitable to generate response message, and described response message is inform the information that the current second packet packet process of described client completes;
Response message sending module, is suitable to described response message is returned described client.
For the server example of Fig. 6, due to itself and the embodiment of the method basic simlarity of Fig. 2,4, so what describe is fairly simple, relevant part illustrates referring to the part of embodiment of the method.
Not intrinsic to any certain computer, virtual system or miscellaneous equipment relevant in algorithm and the display of this offer. Various general-purpose systems can also with use based on together with this teaching. As described above, the structure constructed required by this kind of system is apparent from. Additionally, the present invention is also not for any certain programmed language. It is understood that, it is possible to utilize various programming language to realize the content of invention described herein, and the description above language-specific done is the preferred forms in order to disclose the present invention.
In description mentioned herein, describe a large amount of detail. It is to be appreciated, however, that embodiments of the invention can be put into practice when not having these details. In some instances, known method, structure and technology it are not shown specifically, in order to do not obscure the understanding of this description.
Similarly, it is to be understood that, one or more in order to what simplify that the disclosure helping understands in each inventive aspect, herein above in the description of the exemplary embodiment of the present invention, each feature of the present invention is grouped together in single embodiment, figure or descriptions thereof sometimes. But, the method for the disclosure should be construed to and reflect an intention that namely the present invention for required protection requires feature more more than the feature being expressly recited in each claim. More precisely, as the following claims reflect, inventive aspect is in that all features less than single embodiment disclosed above. Therefore, it then follows claims of detailed description of the invention are thus expressly incorporated in this detailed description of the invention, wherein each claim itself as the independent embodiment of the present invention.
Those skilled in the art are appreciated that, it is possible to carry out the module in the equipment in embodiment adaptively changing and they being arranged in one or more equipment different from this embodiment. Module in embodiment or unit or assembly can be combined into a module or unit or assembly, and multiple submodule or subelement or sub-component can be put them in addition. Except at least some in such feature and/or process or unit excludes each other, it is possible to adopt any combination that all processes or the unit of all features disclosed in this specification (including adjoint claim, summary and accompanying drawing) and so disclosed any method or equipment are combined. Unless expressly stated otherwise, each feature disclosed in this specification (including adjoint claim, summary and accompanying drawing) can be replaced by the alternative features providing purpose identical, equivalent or similar.
In addition, those skilled in the art it will be appreciated that, although embodiments more described herein include some feature included in other embodiments rather than further feature, but the combination of the feature of different embodiment means to be within the scope of the present invention and form different embodiments. Such as, in the following claims, the one of any of embodiment required for protection can mode use in any combination.
The all parts embodiment of the present invention can realize with hardware, or realizes with the software module run on one or more processor, or realizes with their combination. It will be understood by those of skill in the art that can use in practice microprocessor or digital signal processor (DSP) realize the file based on distributed file system according to embodiments of the present invention read, the some or all functions of some or all parts in the process equipment that writes. The present invention is also implemented as part or all the equipment for performing method as described herein or device program (such as, computer program and computer program). The program of such present invention of realization can store on a computer-readable medium, or can have the form of one or more signal. Such signal can be downloaded from internet website and obtain, or provides on carrier signal, or provides with any other form.
The present invention will be described rather than limits the invention to it should be noted above-described embodiment, and those skilled in the art can design alternative embodiment without departing from the scope of the appended claims. In the claims, any reference marks that should not will be located between bracket is configured to limitations on claims. Word " comprises " and does not exclude the presence of the element or step not arranged in the claims. Word "a" or "an" before being positioned at element does not exclude the presence of multiple such element. The present invention by means of including the hardware of some different elements and can realize by means of properly programmed computer. In the unit claim listing some devices, several in these devices can be through same hardware branch and specifically embody. Word first, second and third use do not indicate that any order. Can be title by these word explanations.
Embodiment of the invention discloses that A1, a kind of file reading based on distributed file system, described method includes:
Back end server receives the data access request that client sends, and described data access request includes the information of the target data block that needs read;
The target data block of the information matches of acquisition and described target data block, described target data block includes identification information and the first data message, and described identification information includes verification attribute information;
Described verification attribute information and described first data message are sent to described client.
A2, method according to A1, described first data message includes verification and the data checksum of fragment data chunk and correspondence, wherein, described fragment data chunk and described verification and data checksum are stored in the same storage medium of disk.
A3, method according to A2, the step described verification attribute information and described first data message sent to described client includes:
From the identification information of described target data block, read described verification attribute information send to described client;
From described storage medium, read the default fragment data chunk of size and the verification of correspondence and data checksum sends to described client.
A4, method according to A1, the information of described target data block includes target data block ID, the timestamp GenerationStamp of target data block, and the step of described acquisition and the target data block of the information matches of described target data block includes:
According to the timestamp GenerationStamp of described target data block ID and described target data block, the data block of storage in traversal disk, described data block includes data block ID and data block timestamp;
Search described data block ID identical with described target data block ID, and the described data block timestamp data block identical with the timestamp GenerationStamp of described target data block is as target data block.
A5, method according to A1 or A2 or A3, described verification attribute information includes verification and version Inlinechecksumversion, verification and type checksumtype, and each verification and byte number bytesPerChecksum.
A6, method according to A3, the information of described target data block includes the block bias internal amount startOffset of target data block and the length of target data block, described presetting is sized to the size of the first packet packet, and described reading from described storage medium presets the fragment data chunk of size and the verification of correspondence and step that data checksum sends to described client includes:
Sub-step S11, creates the first packet packet;
Sub-step S12, calculates original position according to the block bias internal amount startOffset of described target data block;
Sub-step S13, from described original position, described storage medium is successively read each fragment data chunk and with in described verification corresponding for fragment data chunk and data checksum to described first packet packet, until fill up described first packet packet;
Sub-step S14, sends described first packet packet to client;
Sub-step S15, repeats above-mentioned sub-step S11 to sub-step S14, until sending the size length equal to described target data block of the data to client.
A7, method according to A2 or A3 or A5, after described verification and data checksum position in described storage medium are positioned at the storage position of the fragment data chunk of correspondence, before being positioned at the storage position of next fragment data chunk of the fragment data chunk of described correspondence.
A8, method according to A1, before described acquisition with the step of the target data block of the information matches of described target data block, also include:
Set up according to described data access request and be connected with the socket of described client.
Embodiments of the invention also disclose B9, a kind of file wiring method based on distributed file system, and described method includes:
Back end server receives the second packet packet, described second packet packet of client transmission and includes target data block ID, verification attribute information and the second data message;
According to the target data block that described target data block ID location is corresponding;
Described target data block ID adds described verification attribute information, generates the identification information of described target data block;
Described second data message is write in described target data block.
B10, method according to B9, described second data message includes multiple fragment data chunk, and respectively with the plurality of verification corresponding for fragment data chunk and data checksum, described by the step that described second data message writes in described target data block be:
Write in described target data block corresponding for target data block ID by described fragment data chunk and with described verification corresponding for fragment data chunk and data checksum, wherein, described fragment data chunk and being stored in the same storage medium of disk with described verification corresponding for fragment data chunk and data checksum.
B11, method according to B9 or B10, described verification attribute information includes verification and version Inlinechecksumversion, verification and type checksumtype, and each verification and byte number bytesPerChecksum.
B12, method according to B9 or B10, described described second data message write the step in described target data block after, also include:
Described second packet packet is transferred in next the back end server in the node queue previously generated by current data node server.
B13, method according to B12, also include:
Generating response message, described response message is inform the information that the current second packet packet process of described client completes;
Described response message is returned described client.
Embodiments of the invention also disclose C14, a kind of node server, and described server includes:
Request receiver module, is suitable to receive the data access request that client sends, and described data access request includes the information of the target data block that needs read;
Locating module, is suitable to obtain the target data block of the information matches with described target data block, and described target data block includes identification information and the first data message, and described identification information includes verification attribute information;
Data transmission blocks, is suitable to send to described client described verification attribute information and described first data message.
C15, node server according to C14, described first data message includes verification and the data checksum of fragment data chunk and correspondence, wherein, described fragment data chunk and described verification and data checksum are stored in the same storage medium of disk.
C16, method according to C15, data transmission blocks includes:
Attribute information reading submodule, is suitable to read described verification attribute information from the identification information of described target data block and sends to described client;
Digital independent submodule, is suitable to the default fragment data chunk of size of reading from described storage medium and the verification of correspondence and data checksum sends to described client.
C17, node server according to C14, the information of described target data block includes target data block ID, the timestamp GenerationStamp of target data block, and described locating module is further adapted for:
According to the timestamp GenerationStamp of described target data block ID and described target data block, the data block of storage in traversal disk, described data block includes data block ID and data block timestamp;
Search described data block ID identical with described target data block ID, and the described data block timestamp data block identical with the timestamp GenerationStamp of described target data block is as target data block.
C18, node server according to C14 or C15 or C16, described verification attribute information includes verification and version Inlinechecksumversion, verification and type checksumtype, and each verification and byte number bytesPerChecksum.
C19, node server according to C16, the information of described target data block includes the block bias internal amount startOffset of target data block and the length of target data block, the described default size being sized to the first packet packet, described data read module is further adapted for:
Create the first packet packet;
Original position is calculated according to the block bias internal amount startOffset of described target data block;
From described original position, described storage medium is successively read each fragment data chunk and with in described verification corresponding for fragment data chunk and data checksum to described first packet packet, until fill up described first packet packet;
Described first packet packet is sent to client.
C20, node server according to C15 or C16 or C18, after described verification and data checksum position in described storage medium are positioned at the storage position of the fragment data chunk of correspondence, before being positioned at the storage position of next fragment data chunk of the fragment data chunk of described correspondence.
C21, node server according to C14, also include:
Connection establishment module, is suitable to set up according to described data access request be connected with the socket of described client.
Embodiments of the invention also disclose D22, a kind of node server, and described node server includes:
Data reception module, the second packet packet, described second packet packet that are suitable to receive client transmission include target data block ID, verification attribute information and the second data message;
Data block locating module, is suitable to the target data block corresponding according to described target data block ID location;
Identification information generation module, is suitable to add described verification attribute information in described target data block ID, generates the identification information of described target data block;
Writing module, is suitable to write in described target data block described second data message.
D23, node server according to D22, described second data message includes multiple fragment data chunk, and respectively with the plurality of verification corresponding for fragment data chunk and data checksum, said write module is further adapted for:
Write in described target data block corresponding for target data block ID by described fragment data chunk and with described verification corresponding for fragment data chunk and data checksum, wherein, described fragment data chunk and being stored in the same storage medium of disk with described verification corresponding for fragment data chunk and data checksum.
D24, node server according to D22 or D23, described verification attribute information includes verification and version Inlinechecksumversion, verification and type checksumtype, and each verification and byte number bytesPerChecksum.
D25, node server according to D11 or D13, also include:
Data transfer module, is suitable to current data node server and is transferred in next the back end server in the node queue previously generated by described second packet packet.
D26, node server according to D25, also include:
Response message generation module, is suitable to generate response message, and described response message is inform the information that the current second packet packet process of described client completes;
Response message sending module, is suitable to described response message is returned described client.
Claims (10)
1., based on a file reading for distributed file system, described method includes:
Back end server receives the data access request that client sends, and described data access request includes the information of the target data block that needs read;
The target data block of the information matches of acquisition and described target data block, described target data block includes identification information and the first data message, and described identification information includes verification attribute information;
Described verification attribute information and described first data message are sent to described client.
2. method according to claim 1, it is characterized in that, described first data message includes verification and the data checksum of fragment data chunk and correspondence, and wherein, described fragment data chunk and described verification and data checksum are stored in the same storage medium of disk.
3. method according to claim 2, it is characterised in that the step that described verification attribute information and described first data message send extremely described client is included:
From the identification information of described target data block, read described verification attribute information send to described client;
From described storage medium, read the default fragment data chunk of size and the verification of correspondence and data checksum sends to described client.
4. the method according to claim 1 or 2 or 3, it is characterised in that described verification attribute information includes verification and version Inlinechecksumversion, verification and type checksumtype and each verification and byte number bytesPerChecksum.
5. method according to claim 3, it is characterized in that, the information of described target data block includes the block bias internal amount startOffset of target data block and the length of target data block, described presetting is sized to the size of the first packet packet, and described reading from described storage medium presets the fragment data chunk of size and the verification of correspondence and step that data checksum sends to described client includes:
Sub-step S11, creates the first packet packet;
Sub-step S12, calculates original position according to the block bias internal amount startOffset of described target data block;
Sub-step S13, from described original position, described storage medium is successively read each fragment data chunk and with in described verification corresponding for fragment data chunk and data checksum to described first packet packet, until fill up described first packet packet;
Sub-step S14, sends described first packet packet to client;
Sub-step S15, repeats above-mentioned sub-step S11 to sub-step S14, until sending the size length equal to described target data block of the data to client.
6., based on a file wiring method for distributed file system, described method includes:
Back end server receives the second packet packet, described second packet packet of client transmission and includes target data block ID, verification attribute information and the second data message;
According to the target data block that described target data block ID location is corresponding;
Described target data block ID adds described verification attribute information, generates the identification information of described target data block;
Described second data message is write in described target data block.
7. method according to claim 6, it is characterized in that, described second data message includes multiple fragment data chunk, and respectively with the plurality of verification corresponding for fragment data chunk and data checksum, described by the step that described second data message writes in described target data block be:
Write in described target data block corresponding for target data block ID by described fragment data chunk and with described verification corresponding for fragment data chunk and data checksum, wherein, described fragment data chunk and being stored in the same storage medium of disk with described verification corresponding for fragment data chunk and data checksum.
8. the method according to claim 6 or 7, it is characterised in that described verification attribute information includes verification and version Inlinechecksumversion, verification and type checksumtype and each verification and byte number bytesPerChecksum.
9. a node server, described server includes:
Request receiver module, is suitable to receive the data access request that client sends, and described data access request includes the information of the target data block that needs read;
Locating module, is suitable to obtain the target data block of the information matches with described target data block, and described target data block includes identification information and the first data message, and described identification information includes verification attribute information;
Data transmission blocks, is suitable to send to described client described verification attribute information and described first data message.
10. a node server, described node server includes:
Data reception module, the second packet packet, described second packet packet that are suitable to receive client transmission include target data block ID, verification attribute information and the second data message;
Data block locating module, is suitable to the target data block corresponding according to described target data block ID location;
Identification information generation module, is suitable to add described verification attribute information in described target data block ID, generates the identification information of described target data block;
Writing module, is suitable to write in described target data block described second data message.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410602747.4A CN105630808A (en) | 2014-10-31 | 2014-10-31 | Distributed file system based file reading and writing method and node server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410602747.4A CN105630808A (en) | 2014-10-31 | 2014-10-31 | Distributed file system based file reading and writing method and node server |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105630808A true CN105630808A (en) | 2016-06-01 |
Family
ID=56045758
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410602747.4A Pending CN105630808A (en) | 2014-10-31 | 2014-10-31 | Distributed file system based file reading and writing method and node server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105630808A (en) |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106375382A (en) * | 2016-08-26 | 2017-02-01 | 苏州蓝海彤翔系统科技有限公司 | Data distribution method and system, control server and node client |
| CN107765994A (en) * | 2016-08-15 | 2018-03-06 | 三星电子株式会社 | The method that data erasing is performed inside intelligence memory device |
| CN108021333A (en) * | 2016-11-03 | 2018-05-11 | 阿里巴巴集团控股有限公司 | The system of random read-write data, device and method |
| WO2018107460A1 (en) * | 2016-12-16 | 2018-06-21 | 华为技术有限公司 | Object-based copying method and apparatus, and object-based storage device |
| WO2019080370A1 (en) * | 2017-10-25 | 2019-05-02 | 华为技术有限公司 | Data reading and writing method and apparatus, and storage server |
| CN109976668A (en) * | 2019-03-14 | 2019-07-05 | 北京达佳互联信息技术有限公司 | Data-erasure method, data deletion apparatus and computer readable storage medium |
| CN110399394A (en) * | 2018-04-16 | 2019-11-01 | 北京京东尚科信息技术有限公司 | The data traversal method, apparatus and calculate node of calculate node |
| CN110795407A (en) * | 2019-10-14 | 2020-02-14 | 华东计算技术研究所（中国电子科技集团公司第三十二研究所） | File random writing method and system suitable for distributed file system |
| CN110989922A (en) * | 2019-10-28 | 2020-04-10 | 烽火通信科技股份有限公司 | Distributed data storage method and system |
| CN111190537A (en) * | 2019-12-10 | 2020-05-22 | 优刻得科技股份有限公司 | Method and system for managing sequential storage disks in write-addition scene |
| CN112688905A (en) * | 2019-10-18 | 2021-04-20 | 成都华为技术有限公司 | Data transmission method, device, client, server and storage medium |
| CN114281267A (en) * | 2021-12-30 | 2022-04-05 | 西北工业大学 | Data migration method and device between distributed storage systems |
| CN114327283A (en) * | 2021-12-30 | 2022-04-12 | 西北工业大学 | Data migration method in distributed storage system and distributed storage system |
| CN114363321A (en) * | 2021-12-30 | 2022-04-15 | 支付宝(杭州)信息技术有限公司 | File transmission method, equipment and system |
| CN116700632A (en) * | 2023-08-07 | 2023-09-05 | 湖南中盈梦想商业保理有限公司 | High-reliability financial information data storage method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6637007B1 (en) * | 2000-04-28 | 2003-10-21 | Network Appliance, Inc. | System to limit memory access when calculating network data checksums |
| US20090125753A1 (en) * | 2007-11-13 | 2009-05-14 | Anton Joseph Kryka | Handling of data storage within a flash media device |
| CN102025444A (en) * | 2009-09-09 | 2011-04-20 | 腾讯科技（深圳）有限公司 | Resource publishing method, resource downloading method and data verification system |
| CN103984607A (en) * | 2013-02-08 | 2014-08-13 | 华为技术有限公司 | Distributed storage method, device and system |
-
2014
- 2014-10-31 CN CN201410602747.4A patent/CN105630808A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6637007B1 (en) * | 2000-04-28 | 2003-10-21 | Network Appliance, Inc. | System to limit memory access when calculating network data checksums |
| US20090125753A1 (en) * | 2007-11-13 | 2009-05-14 | Anton Joseph Kryka | Handling of data storage within a flash media device |
| CN102025444A (en) * | 2009-09-09 | 2011-04-20 | 腾讯科技（深圳）有限公司 | Resource publishing method, resource downloading method and data verification system |
| CN103984607A (en) * | 2013-02-08 | 2014-08-13 | 华为技术有限公司 | Distributed storage method, device and system |
Non-Patent Citations (1)
| Title |
|---|
| 阿涵: ""HDFS关键数据结构详解"", 《HTTP://BLOG.SINA.COM.CN/S/BLOG_3FE961AE01019CIX.HTML》 * |
Cited By (26)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107765994A (en) * | 2016-08-15 | 2018-03-06 | 三星电子株式会社 | The method that data erasing is performed inside intelligence memory device |
| CN107765994B (en) * | 2016-08-15 | 2023-01-03 | 三星电子株式会社 | Method for executing data erasure inside intelligent memory device |
| CN106375382A (en) * | 2016-08-26 | 2017-02-01 | 苏州蓝海彤翔系统科技有限公司 | Data distribution method and system, control server and node client |
| CN108021333A (en) * | 2016-11-03 | 2018-05-11 | 阿里巴巴集团控股有限公司 | The system of random read-write data, device and method |
| WO2018107460A1 (en) * | 2016-12-16 | 2018-06-21 | 华为技术有限公司 | Object-based copying method and apparatus, and object-based storage device |
| CN108475254A (en) * | 2016-12-16 | 2018-08-31 | 华为技术有限公司 | Object clone method, device and object storage device |
| WO2019080015A1 (en) * | 2017-10-25 | 2019-05-02 | 华为技术有限公司 | Data reading and writing method and device, and storage server |
| US11397668B2 (en) | 2017-10-25 | 2022-07-26 | Huawei Technologies Co., Ltd. | Data read/write method and apparatus, and storage server |
| WO2019080370A1 (en) * | 2017-10-25 | 2019-05-02 | 华为技术有限公司 | Data reading and writing method and apparatus, and storage server |
| CN110399394A (en) * | 2018-04-16 | 2019-11-01 | 北京京东尚科信息技术有限公司 | The data traversal method, apparatus and calculate node of calculate node |
| CN109976668A (en) * | 2019-03-14 | 2019-07-05 | 北京达佳互联信息技术有限公司 | Data-erasure method, data deletion apparatus and computer readable storage medium |
| CN110795407A (en) * | 2019-10-14 | 2020-02-14 | 华东计算技术研究所（中国电子科技集团公司第三十二研究所） | File random writing method and system suitable for distributed file system |
| CN110795407B (en) * | 2019-10-14 | 2022-06-10 | 华东计算技术研究所（中国电子科技集团公司第三十二研究所） | File random writing method and system suitable for distributed file system |
| CN112688905A (en) * | 2019-10-18 | 2021-04-20 | 成都华为技术有限公司 | Data transmission method, device, client, server and storage medium |
| CN112688905B (en) * | 2019-10-18 | 2023-04-18 | 华为云计算技术有限公司 | Data transmission method, device, client, server and storage medium |
| CN110989922A (en) * | 2019-10-28 | 2020-04-10 | 烽火通信科技股份有限公司 | Distributed data storage method and system |
| CN110989922B (en) * | 2019-10-28 | 2023-05-26 | 烽火通信科技股份有限公司 | Distributed data storage method and system |
| CN111190537A (en) * | 2019-12-10 | 2020-05-22 | 优刻得科技股份有限公司 | Method and system for managing sequential storage disks in write-addition scene |
| CN111190537B (en) * | 2019-12-10 | 2023-08-25 | 优刻得科技股份有限公司 | Method and system for managing sequential storage disk in additional writing scene |
| CN114363321A (en) * | 2021-12-30 | 2022-04-15 | 支付宝(杭州)信息技术有限公司 | File transmission method, equipment and system |
| CN114327283A (en) * | 2021-12-30 | 2022-04-12 | 西北工业大学 | Data migration method in distributed storage system and distributed storage system |
| CN114281267A (en) * | 2021-12-30 | 2022-04-05 | 西北工业大学 | Data migration method and device between distributed storage systems |
| CN114327283B (en) * | 2021-12-30 | 2024-03-12 | 西北工业大学 | Data migration method in distributed storage system and distributed storage system |
| CN114281267B (en) * | 2021-12-30 | 2024-04-26 | 西北工业大学 | Data migration method and device between distributed storage systems |
| CN116700632A (en) * | 2023-08-07 | 2023-09-05 | 湖南中盈梦想商业保理有限公司 | High-reliability financial information data storage method |
| CN116700632B (en) * | 2023-08-07 | 2023-10-24 | 湖南中盈梦想商业保理有限公司 | High-reliability financial information data storage method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105630808A (en) | Distributed file system based file reading and writing method and node server | |
| CN111164585B (en) | Performing in-memory rank analysis queries on externally resident data | |
| US11281531B2 (en) | Serial storage node processing of data functions | |
| US10795817B2 (en) | Cache coherence for file system interfaces | |
| US10852961B2 (en) | Overlapping write detection and processing for sync replication | |
| US9830345B1 (en) | Content-addressable data storage | |
| US11321291B2 (en) | Persistent version control for data transfer between heterogeneous data stores | |
| EP2631819A1 (en) | Database controller, method, and system for storing encoded triples | |
| US11226944B2 (en) | Cache management | |
| CN103150394A (en) | Distributed file system metadata management method facing to high-performance calculation | |
| US10909086B2 (en) | File lookup in a distributed file system | |
| US11151081B1 (en) | Data tiering service with cold tier indexing | |
| US11188505B2 (en) | System and method for batch database modification | |
| CN103617199A (en) | Data operating method and data operating system | |
| CN104778192A (en) | Representing directory structure in content-addressable storage systems | |
| US20230120592A1 (en) | Query Generation and Processing System | |
| EP4150484A1 (en) | Efficient indexing for querying arrays in databases | |
| US20210286806A1 (en) | Personal information indexing for columnar data storage format | |
| US20200250232A1 (en) | Partial file system instances | |
| US11269930B1 (en) | Tracking granularity levels for accessing a spatial index | |
| CN108256019A (en) | Database key generation method, device, equipment and its storage medium | |
| US11106667B1 (en) | Transactional scanning of portions of a database | |
| US11055018B2 (en) | Parallel storage node processing of data functions | |
| US11314593B2 (en) | Storage node processing of data functions using overlapping symbols | |
| US20230122781A1 (en) | Low-Latency Buffer Storage Of Static Datasets For Query Operation Optimization |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160601 |