JP6778795B2 - Methods, devices and systems for storing data - Google Patents

Description

Embodiments of the present invention relate to the technical field of distributed object storage, specifically to methods, devices and systems for storing data.

A distributed storage system typically stores data in a distributed manner across a plurality of independent devices. Traditional network storage systems use a centralized storage server to store all the data. The storage server at this time becomes a bottleneck of system performance and also becomes a focus of reliability and safety, and cannot meet the needs of large-scale storage applications. A distributed network storage system often uses a scalable system structure, uses multiple storage servers to share a storage load, and uses a location server to determine the location of storage information. This not only improves system reliability, availability, and access efficiency, but also facilitates expansion.

Embodiments of the present invention provide methods, devices and systems for storing data.

In a first aspect, embodiments of the present invention provide a method for storing data. The method is used for a stand-alone storage engine of distributed object storage, a storage file is configured on the disk of the stand-alone storage engine, the storage space of the storage file is divided into at least two data blocks, and the at least two data. Using a link list structure between blocks, the method determines the type of the storage record based on the size of the data in the storage record and the current storage information of the type in the storage file. By retrieving, the current storage information includes information on the currently allocated data block and information on the currently stored record of the data block and is stored in the same type of storage file. The size of the data in the records is the same, the size of the data in the records stored in different types is different, and the remaining storage space in the currently allocated data block based on the current storage information. In response to determining whether or not exists and determining that there is no remaining storage space, a new data block is assigned to the type and the storage target record is stored in the new data block. Including to do.

In some embodiments, the method for storing data is such that the remaining storage space is not smaller than the size of the data in the storage target record in response to the determination that there is the remaining storage space. In response to further determining whether or not, and determining that the remaining storage space is smaller than the size of the data in the storage target record, some data in the storage target record is stored in the remaining storage. It further includes storing in a space and allocating a new data block to the type to store the rest of the data in the storage target record in the new data block.

In some embodiments, the storage space of the at least two data blocks is the same, and the size of the data in the records stored in different types is all an integral multiple of a predetermined number.

In some embodiments, the method for storing data is to update the current storage information of the type, generate the stored location information of the storage target record, and output the location information. Further included, the location information includes at least one of the type of record, the identifier of the record, and the identifier of the data block in which the record is located.

In some embodiments, at least two storage files are configured on the disk, and index information for each storage file is stored in a directory on the disk.

In some embodiments, the method for storing data is to retrieve the location information of the read target record from the directory and, based on the location information of the read target record, the type of the read target record and Determine the position offset in the located data block of the corresponding storage file, determine the length of the read target record based on the type of the read target record, and determine the position displayed by the position offset. It further includes reading data having a length corresponding to the length of the read target record, and outputting the read data as the read target record.

In some embodiments, the method for storing data is to search for the location information of the record to be deleted from the directory and to delete the deletion in the corresponding storage file based on the location information of the record to be deleted. Acquiring the current storage information of the type to which the target record belongs, reading the last record from the currently stored record based on the acquired current storage information, and deleting the read record. It further includes transferring and storing the position of the record, clearing the data in the pre-transfer data block of the read record, and correcting the position information of the read record.

In some embodiments, the method for storing the data after clearing the data in the pre-transfer data block of the read record is to the data block where the read record was located before the transfer. It further includes determining if the data exists and, if the data block does not exist, retrieving the data block for reassignment.

In some embodiments, a method for storing the data, based on the size of the data in the storage target record, before determining the type of the storage target record, is obtained by dividing the storage waiting object. For the sub-object in at least one sub-object, the sub-object is coded to obtain a copy, and the explanatory information and data of the copy are arranged to obtain the sub-object. It further includes generating a record to be stored.

In some embodiments, generating a storage target record for the sub-object is the same as the size of the data in the duplicate after arraying is one of the sizes of data corresponding to each of the types in the storage file. If the size of the data of the duplicate is different from the size of the data corresponding to each of the types and is smaller than the size of the data corresponding to some of the types, the size of the duplicate is determined. Fill the back of the data with zeros so that the size of the duplicate data after filling with zeros is the same as the size of the data corresponding to the target type, and one storage target record of the sub-object is generated. If the size of the data of the duplicate is larger than the maximum value of the size of the data corresponding to each of the types, the size of the data of each duplicate after division is increased by dividing the duplicate and filling it with zeros. The target type includes the creation of at least two storage target records of the sub-object, each of which is the same as one of the sizes of data corresponding to each of the above types. Of the types, the type has the smallest corresponding data size.

In a second aspect, embodiments of the present invention provide a device for storing data. The device is configured as a stand-alone storage engine for distributed object storage, storage files are configured on the disk of the stand-alone storage engine, the storage space for the storage files is divided into at least two data blocks, and the at least two data. Using a link list structure between blocks, the device comprises a type determination unit configured to determine the type of the storage target record based on the size of the data in the storage target record, and said in the storage file. A search unit configured to search for the type of current storage information, the current storage information being the information of the currently allocated data block and the information of the currently stored record of the data block. Based on the search unit and the current storage information, including, the data size of the records stored in the same type of the storage file is the same, but the data size of the records stored in different types is different. In response to the space determination unit configured to determine if there is any remaining storage space in the currently allocated data block and the determination that there is no remaining storage space. It comprises a first allocation unit configured to allocate a new data block to the type and store the storage target record in the new data block.

In some embodiments, the device for storing the data does not have the remaining storage space smaller than the size of the data in the storage target record in response to the determination that there is remaining storage space. A sizing unit that is placed to further determine whether or not, and some data in the storage record in response to the determination that the remaining storage space is smaller than the size of the data in the storage record. It further comprises a second allocation unit, which stores in the remaining storage space, allocates new data blocks to the type, and is arranged to store the remaining data in the storage target record.

In some embodiments, the storage space of at least two data blocks is the same, and the size of the data in the records stored in different types is all an integral multiple of a given number.

In some embodiments, the device for storing the data is arranged to update the current storage information of the type, generate the location information after the storage target record is stored, and output the location information. It further comprises a position generation unit, wherein the position information includes at least one of a record type, a record identifier, and an identifier of the data block in which the record is located.

In some embodiments, at least two of the storage files are configured on the disk, and the index information of each of the storage files is stored in the directory of the disk.

In some embodiments, the device for storing the data is a first position search unit arranged to search the position information of the record to be read from the directory, and the device for storing the data is based on the position information of the record to be read. Determines the length of the record to be read based on the type of record to be read and the type of record to be read, and the confirmation unit that is placed to determine the position offset in the located data block of the corresponding storage file. Further, a read unit arranged to read data whose length is the length of the read target record, starting from the position indicated by the position offset, and to output the read data as the read target record. Be prepared.

In some embodiments, the device for storing the data is based on a second position search unit arranged to search the position information of the record to be deleted from the directory and the position information of the record to be deleted. The last record from the currently stored record is based on the acquisition unit that is placed to acquire the current storage information of the type to which it belongs in the corresponding storage file of the record to be deleted and the current storage information acquired. The transfer unit arranged to transfer and store the read and read records to the position of the record to be deleted, and the data in the data block before transfer of the read records are cleared and read. It further includes a correction unit arranged to correct the position information of the record.

In some embodiments, the device for storing the data determines if the data still exists in the data block located before the read record is transferred and the data is in the data block. It further comprises a recovery unit that is arranged to recover the data block for reassignment if it is not stored.

In some embodiments, the device for storing the data is such that the sub-object in at least one sub-object obtained by dividing the storage waiting object is coded to obtain a copy. Further includes a coding unit arranged in the above, and a record generation unit arranged so as to perform an arrangement process on the explanatory information and data of the duplicate to generate a storage target record of the sub-object.

In some embodiments, the record generation unit determines whether the size of the sequenced duplicate data is the same as one of the sizes of data corresponding to each type in the storage file. If the size of the data of the duplicate and the fixed subsystem placed in is different from the size of the data corresponding to each type and smaller than the size of the data corresponding to some types, the data of the duplicate The first generation sub that fills the back with zeros, makes the size of the data of the duplicate after filling with zeros the same as the size of the data corresponding to the target type, and generates one storage target record of the duplicate sub-object. The target type is the unit, the first generation subsystem, which is the type with the smallest corresponding data size among the some types, and the data whose data size of the duplicate corresponds to each type. If it is larger than the maximum size, the duplicates are split and zero-padded so that the size of the data in each duplicate after being split is the same as one of the sizes of data corresponding to each type. It comprises a second generation subsystem arranged to generate at least two storage target records of the sub-object.

In a third aspect, an embodiment of the present invention comprises a third subsystem in which the stand-alone storage engine according to any one of the first subsystem, the second subsystem and the first embodiment is installed. Provide a system. The first subsystem receives a storage request including a storage-waiting object transmitted by a user, divides the storage-waiting object into at least one sub-object, and the storage-waiting object and the at least one sub-object. The correspondence between the two subsystems is transmitted to the second subsystem, and the at least one subsystem is transmitted to the third subsystem, and the second subsystem is the storage waiting object and the at least one. The third subsystem is configured to store the correspondence with the sub-object in a list, and the sub-object in the at least one sub-object is coded and arranged, and the sub-object is processed. The storage target record of is generated, and the generated storage target record is stored.

In some embodiments, the third subsystem is further configured to transmit response information to indicate the completion of data storage to the first subsystem, which further configures the first subsystem to further display the response. When the information is received, the query identifier of the storage waiting object is generated, and the query identifier is configured to be fed back to the user.

In some embodiments, the first subsystem is further configured to receive a read request including the query identifier sent by the user and send the query identifier in the read request to the second subsystem. The second subsystem is further configured to acquire the sub-object list corresponding to the object indicated by the query identifier in the read request and send the sub-object list to the third subsystem. The third subsystem reads the corresponding record based on the sub-object list, analyzes the read record to acquire the object data, and the first subsystem feeds the object data back to the user. Is further configured to transmit the object data to the first subsystem.

In a fourth aspect, an embodiment of the invention comprises one or more processors and a storage device for storing one or more programs, wherein the one or more programs are the one or more. When executed by a plurality of processors, said one or more processors provide an electronic device that realizes the method according to any embodiment of the first aspect.

In a fifth aspect, the embodiment of the present invention is a computer-readable medium in which the method according to any embodiment of the first aspect is realized when a computer program is stored and the program is executed by a processor. I will provide a.

The methods, devices and systems for storing data provided by embodiments of the present invention can determine the type of storage target record based on the size of the data in the storage target record. It allows you to retrieve the type's current storage information in a storage file. The current storage information can include information on the currently allocated data block and record information currently stored in the data block. Here, the size of the data of the records stored in the same type of the storage files is the same, and the size of the data of the records stored in different types is different. In addition, based on the current storage information, it is possible to determine if there is any remaining storage space in the currently allocated data block. A new block of data can be assigned to a type if it is determined that there is no remaining storage space. In addition, the storage target record can be stored in a new data block. As a result, the occurrence of space debris can be reduced and the space utilization rate of the disk can be improved. It is also advantageous for improving the read / write performance of the disk.

The detailed description given to the non-limiting examples with reference to the drawings further reveals other features, objectives and advantages of the present invention.
It is an exemplary system architecture diagram to which one embodiment of the present invention is applied. It is a flowchart of one Example of the method for storing the data which concerns on this invention. It is a flowchart of another embodiment of the method for storing the data which concerns on this invention. It is a flowchart of another embodiment of the method for storing the data which concerns on this invention. It is a logical structure schematic diagram of one Example of the storage file in this invention. It is a structural schematic diagram of one Example of a record in this invention. It is a structural schematic diagram of one Example of a disk in this invention. It is a structural schematic diagram of one Example of the apparatus for storing the data which concerns on this invention. It is a timing diagram of one Example of the system for storing the data which concerns on this invention. It is a structural schematic diagram of the computer system for realizing the electronic device of the Example of this invention.

Hereinafter, the present invention will be described in more detail with reference to the drawings and examples. It is understood that the specific examples described herein are not limited to the invention, but are merely used to interpret the relevant invention. For convenience of explanation, only the parts related to the invention are shown in the drawings.

It should be explained that, if there is no contradiction, the examples in the present invention and the features in the examples can be combined with each other. Hereinafter, the present invention will be described in detail with reference to the drawings in connection with examples.

FIG. 1 shows an exemplary system architecture 100 of methods, devices and systems for storing data to which examples of the present invention are applied.

As shown in FIG. 1, the system architecture 100 can include terminals 101, 102, 103, a network 104, and a server 105. The network 104 is used as a medium for providing a communication link between terminals 101, 102, 103 and the server 105. The network 104 can include various connections such as wired, wireless communication links or fiber optic cables.

The user can use the terminals 101, 102, and 103 to interact with the server 105 via the network 104 to send and receive messages and the like. Various client applications such as web browsers, shopping applications, video applications, mailboxes and instant messaging tools can be installed on terminals 101, 102 and 103.

Here, the terminals 101, 102, and 103 can be hardware or software. When the terminals 101, 102, and 103 are hardware, they may be various electronic devices having a display screen. The electronic device may be a smartphone, tablet computer, smart TV, e-book reader, MP3 (Moving Picture Experts Group Audio Layer III) player, laptop computer, desktop computer, or the like. However, it is not limited to these. If the terminals 101, 102, 103 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (eg, used to provide distributed services) or as a single software or software module. It is not specifically limited here.

The server 105 can be a server that provides various services, for example, a background server that provides support for various applications installed on terminals 101, 102, 103. The background server can analyze and process the data processing request (for example, data storage request) transmitted by the user via the terminals 101, 102, 103, and the processing result (for example, the storage is completed). The feedback information to be displayed or the position information after data storage can be returned to the terminals 101, 102, 103.

Here, the server 105 can be the same hardware or software. When the server 105 is hardware, it can be realized by a distributed server group composed of a plurality of servers, and can also be realized by one server. When the server 105 is software, it can be implemented by a plurality of software or software modules (eg, used to provide distributed services), and can also be implemented by a single software or software module. It is not specifically limited here.

For example, in the server 105, a plurality of storage files 1051 and 1052 can be configured on a disk (that is, a storage space) for storing data. These storage files can be the actual container for storing data, thereby achieving distributed storage of data.

It should be explained that the method for storing the data provided by the embodiments of the present invention is generally performed by the server 105. Correspondingly, a device for storing data is generally installed in the server 105.

It should be understood that the number of terminals, networks, servers and storage files in the server in FIG. 1 is merely exemplary. Any number of terminals, networks, servers and storage files in the servers can be provided according to the realized demand.

Subsequently, with reference to FIG. 2, a flow 200 of one embodiment of the method for storing data according to the present invention is shown. The method for storing the data can include the following steps.

In step 201, the type of storage target record is determined based on the size of the data in the storage target record.

In this embodiment, the method for storing data can be used for a stand-alone storage engine for distributed object storage. In addition, storage files can be configured on the disk of the stand-alone storage engine. Here, the storage file can be an actual container for storing data. That is, the data is stored in the storage file. Here, the storage space of the storage file can be divided into at least two data blocks. The storage space of a storage file can be further divided into a plurality of storage sections (that is, data blocks). Also, a linked list structure can be used between at least two data blocks to ensure storage space continuity. That is, each data block in the same storage file is linked by the linked list method.

The point to be explained is that the storage capacity of the storage file can be set according to the actual situation, and can be, for example, 32G. Also, the storage space of each data block in the same storage file can be the same (eg, 2M) and can be different. Inside the storage file, the storage target data can be classified and stored based on the size of the storage target data. In addition, only one size of data can be stored in each type. That is, the size of the data stored in the same type can be the same. The size of the data stored in different types can be different. Here, the classification method is not limited by the present invention.

In this embodiment, the execution subject of the method for storing data (eg, the server 105 shown in FIG. 1) is a terminal (eg, shown in FIG. 1) via a wired or wireless connection method. The storage target record can be acquired from the terminal 101, 102, 103) or the cloud. Further, the type of the storage target record is determined by a predetermined classification method based on the data size of the storage target record. For example, the size of the data of the storage target record is 4KB. When the size of the data is larger than 0KB and not larger than 4KB, it belongs to the first type, and in this case, the executing subject can determine the type of the storage target record as the first type. Here, the storage target record can be any data to be stored using the disk distributed object storage technology. The content can include a string of at least one character code, such as a number, letter, alphabet or symbol.

In certain selectable embodiments of this embodiment, for convenience of management, the storage space of at least two data blocks in the same storage file can be the same. The size of the data of records stored in different types of the same storage file can all be an integral multiple of a given number. This helps to simplify the calculation process and improve processing efficiency. As shown in FIG. 5A, the storage space of all data blocks is 2M. Considering that the minimum value of the data size is generally 4KB, the size of the data corresponding to different types are all integer multiples of 4KB. Further, considering the read / write performance of the disk, the maximum value of the data size can be set according to the actual situation, and can be set to, for example, 512 KB. In other words, the size of the record data can be divided into a plurality of types such as 4KB, 8KB ... 512KB inside the storage file.

In step 202, the storage file is retrieved for the type's current storage information, which includes information about the currently allocated data block and information about the currently stored record of the data blocks.

In this embodiment, the execution subject can search the storage file for the current storage information of the type to which the storage target record belongs. Here, the current storage information can include information on the currently allocated data block and record information currently stored in the data block. That is, the information of the data block currently assigned to the type and the record information currently stored in the data block are searched. Here, the information of the data block includes, but is not limited to, at least one of the link order between the data blocks, the identifier of the data block, the storage space of the data block, and the like. The record information includes, but is not limited to, an identifier of the record, the size of the data of the record, the identifier of the data block in which the record is located, and the like. Here, the identifier can be used to uniquely indicate a data block or record, and can be at least one of character codes such as numbers and alphabets. In other words, after retrieving the current storage information for the type, the executor knows what data blocks exist for the type and what records are stored.

It should be explained that records of the same type are generally stored side by side one by one. That is, logically, these records occupy a continuous space as a whole. This can improve the utilization rate of the storage space by preventing the generation of disk fragments.

As an example, assuming that the storage target record of FIG. 5A is X, when the executing subject determines that the type (type) is 128, the current storage information of the type can be searched. That is, the storage space is a 2M data block 37, the number of records is 4, and the data size of each record is 512KB. It can be understood that numeric numbers (1, 2, 3, etc.) are used as record identifiers. In this way, not only can different records of the same type be distinguished, but the storage order of each record can be easily determined.

In step 203, based on the current storage information, it is determined whether or not there is remaining storage space in the currently allocated data block.

In this embodiment, the executing entity has the remaining storage space in the currently allocated data block of the type (that is, the type to which the storage target record belongs) based on the current storage information retrieved in step 202. Whether or not it can be determined. By way of example, as shown in FIG. 5A, the executing entity can determine that there is no remaining storage space in block37 based on the current storage information in type = 128, ie, at 512KB × 4 = 2M. is there.

Here, if there is a remaining storage space in the currently allocated data block, the executing subject can store the storage target record in the remaining storage space. If there is no remaining storage space in the currently allocated data block, the executor can subsequently perform step 204.

Selectably, the executor should further determine if the remaining storage space is not smaller than the size of the data in the storage record if it is determined that the allocated data block has remaining storage space. Can be done. If it is determined that the remaining storage space is smaller than the size of the data in the storage target record, some data in the storage target record can be stored in the remaining storage space. Then, a new data block can be assigned to the type, and the remaining storage target records can be stored in the new data block.

By way of example, as shown in FIG. 5A, the size of the data is 504KB of the storage target record Y. The executing entity can determine that the remaining storage space of block3 is 64KB (4M-504KB × 8) based on the current storage information of type = 126. At this time, the executing subject can allocate one new 2M data block (block4) under the type. In this way, 64KB before the storage target record Y can be stored in block3. In addition, 440 KB after the storage target record Y can be stored in block4. That is, the storage target record Y can be stored in two adjacent data blocks.

In step 204, in response to the determination that there is no remaining storage space, a new data block is assigned to the type and the records to be stored are stored in the new data block.

In this embodiment, the executing entity can allocate a new data block to the type if it is determined that there is no remaining storage space in the currently allocated data block. In addition, the storage target record can be stored in a new data block. As an example, it is the storage target record X of FIG. 5A. Since there is no remaining storage space in block37, the executor can allocate one new block49. At this time, the storage target record X can be stored in the block49.

In the method for storing the data in this embodiment, the storage target records can be classified and stored based on the data size of the storage target records. It is possible to secure continuous storage of data and reduce or prevent the generation of disk fragments. In this way, it helps to improve the utilization rate of the storage space and the read / write performance of the disk. The distributed object storage method will also be enhanced.

In certain selectable embodiments of this embodiment, after storing a record to be stored, the executor can also update the current storage information of the type to which it belongs. Then, the position information after the storage target record is stored can be generated. In addition, the position information can be output. Here, the position information may be for explaining the storage position of the storage target record. For example, the location information can include at least one of a record type, a record identifier, and an identifier of the data block in which the record is located, and the location information is output. Here, the output may be to store and output. For example, the location information can be stored locally in the execution subject or in another electronic device, and for example, the transmission output may be such that the location information is transmitted to a terminal or the like.

As an example, in the case of the storage target record X of FIG. 5A, the position information can be [record_type = 128, record_id = 5, block_id = 49, next_block_id = 0]. It can be understood that the executing subject can search the record X even when the location information is [record_type = 128, record_id = 5] or [record_id = 5, block_id = 49].

The point to be explained is that, in order to ensure that the data size of the records stored in the same type is the same, the execution subject can select the data of the storage target record before storing the storage target record. It can be determined whether the size is the size of the data corresponding to the type. When the size of the data of the storage target record is determined to be the size of the data corresponding to the type, the executing subject can perform the storage process. If the size of the data in the storage record is determined not to be the size of the data corresponding to the type and is generally smaller than the size of the data corresponding to the type, the executing entity is at a predetermined position (eg, front or front) of the storage record The rear surface) can be supplemented with predetermined data (for example, 0 or another character string). As a result, the size of the data of the storage target record after being replenished becomes the size of the data corresponding to the type. After that, the executing subject can store the replenished storage target record.

In some application scenarios, the executing entity can process the data to acquire the storage target record before determining the type of storage target record based on the size of the data in the storage target record. For example, for a sub-object in at least one sub-object obtained by dividing a storage-waiting object, the execution subject codes the sub-object (for example, EC coding) to acquire a copy. After that, the explanatory information and data of the duplicate are subjected to the array processing to generate the storage target record of the sub-object.

The point to be explained is that the process of dividing the storage-waiting object can be completed by the executing subject or by another electronic device. Other electronic devices can transmit at least one sub-object obtained by dividing to the executing subject.

Further, the executing subject analyzes and processes the duplicated copy after the array so that the size of the record data stored in the same type is the same, and the size of the required data (that is, the data corresponding to each type). It is possible to generate a record to be stored (one of the sizes).

Specifically, first, the executing subject can determine whether or not the size of the data of the duplicated copy after the array is the same as one of the sizes of the data corresponding to each type of the storage file. .. When the size of the data of the duplicate is the same as one of the sizes of the data corresponding to each type, the executing subject can directly generate the storage target record of the sub-object.

The size of the duplicate data (eg, 5KB) is all different from the size of the data corresponding to each type (eg, 4KB, 8KB and 12KB), and the size of the data corresponding to some types (eg, 8KB and). If it is smaller than 12KB), fill the end of the duplicate data with zeros. This allows the size of the duplicate data after filling with zeros to be the same as the size of the data corresponding to the target type. Then, one storage target record of the sub-object can be generated. Here, the target type can be the type with the smallest corresponding data size of some of the types (eg, 8KB out of 8KB and 12KB). It should be explained that the present invention is not limited to supplementary positions and supplementary content as long as it does not affect conventional data.

When the size of the data of the duplicate is larger than the maximum value of the size of the data corresponding to each type (for example, 512 KB), the duplicate can be divided and filled with zeros. As a result, the size of the data of each duplicate after division can be made the same as one of the sizes of the data corresponding to each type, and at least two storage target records of the sub-object are generated. For example, the size of the duplicate data is 518KB, which can be divided into two sub-duplicates, 512KB and 6KB, and the 6KB sub-duplicates are filled with zeros to 8KB. Alternatively, it can be divided into three sub-duplicates of 512KB, 4KB and 2KB, and the 2KB sub-duplicates are filled with zeros in 4KB. Alternatively, the duplicate can be divided into multiple sub-duplicates of the corresponding data size based on the size of the data corresponding to the other type, and there is no need to perform the fill operation with zeros.

By way of example, the storage target record can include a field (field) as shown in FIG. 5B. Here, a part of the meta information (Meta information) of the duplicate (hard) can be recorded in the Hard Record (shared record), and the cyclic surplus check code (crc: Cyclic Redundancy Code), length, etc. can be recorded. Can include content. Explanatory information specified by the user can be recorded in Key + meta (key meta). The Binary data can generally be the actual data of the duplicate (hard data). On the back of the data, there can be a field of unfixed length to fill with zeros (ie, field zero). This makes it possible to ensure that each record is an integral multiple of 4 kb.

It can be understood that each storage file can also self-manage the internal storage space in order to reduce the load on the execution subject. For example, one data block management unit (BlockManager) 501 and a plurality of type management units (BlockLinkList) 502 can be installed for one storage file. Here, the data block management unit can be used to allocate and retrieve data blocks. Each type management unit can be for managing one type of storage information. For example, the type management unit of type = 128 in FIG. 5A can store a list of self-assigned data blocks (eg, block37) and the number of records it has entered (eg, 4). In this way, the number recorded can be used to calculate how much storage space is left available in the last block of data (eg, block37). If there is not enough storage space, information can be sent to the data block management unit to indicate that the data block has been allocated. At this time, the data block management unit can assign one new data block (for example, block49) to it. In this way, it is useful to improve the processing efficiency and performance of the entire execution subject.

At this time, the internal structure of each storage file can be constructed by the location information (location) already stored and recorded by oneself. At startup, each storage file can scan all of its records once. This allows you to display all the data blocks you have assigned to BlockManager. In addition, the position information of each record can be restored by BlockLinkList. After completing the scan, the Block Manager will be able to see the allocation information for its data blocks. All types of BlockLinkList will also be able to see information about their data blocks and their current record.

The method for storing the data in this embodiment can determine the type of the storage target record based on the size of the data of the storage target record. This allows you to look up the current storage information for a type in a storage file. The current storage information can include information on the currently allocated data block and record information currently stored in the data block. Here, the size of the data of the records stored in the same type of the storage files is the same, and the size of the data of the records stored in different types is different. In addition, based on the current storage information, it is possible to determine if there is any remaining storage space in the currently allocated data block. If it is determined that there is no remaining storage space, a new block of data can be assigned to the type. In addition, the storage target record can be stored in a new data block. In this way, it is possible to reduce the generation of space fragments and improve the space utilization rate of the disk. It is also advantageous for improving the read / write performance of the disk.

FIG. 3 shows a flow 300 of another embodiment of the method for storing data according to the present invention. In this embodiment, at least two storage files can be configured on the disk. In addition, the index information of each storage file on the disk can be stored in the directory of the disk. In other words, each storage file can contain index information and data information. Here, in the index information, what kind of record and the state of the record are stored inside the storage file are recorded. Data information mainly refers to the actual data stored inside the storage file. That is, for all the records input to the disk, the index information of the record is recorded in the directory. Here, the index information can include the position information of the record.

As an example, the structure of the disc is shown in FIG. 5C. Here, Rocksdb503 can display the directory of the disk, use the key-value (key-value) storage method, and Vlet504 (Vlet_110_3 and Vlet ...) can display the storage file. It should be explained that one disk generally has only one directory example. By storing the index information and the data information separately, it is possible to ensure that the existence or nonexistence of the record can be quickly searched. Selectably, the index information can be stored in full memory, but is limited by the size of the memory. Here, if you store using a directory, you will not be limited by memory, but another IO can occur. At this time, a cache memory can be configured to preferably balance the use of memory with the use of IO.

In this embodiment, the method for storing data can further include the following steps:

In step 301, the position information of the record to be read is searched from the directory.

In this embodiment, the executing subject of the method for storing data (for example, the server 105 shown in FIG. 1) can search the directory for the record to be read. When the record to be read is searched, the location information can be obtained from the directory.

In step 302, based on the position information of the read target record, the position offset of the type of the read target record and the corresponding storage file in the located data block can be determined.

In this embodiment, when the position information includes the record type, the executing subject can determine the type of the read target record based on the position information of the read target record. Alternatively, the executing subject can determine the storage file and the data block in which the record to be read is located based on the location information, thereby determining the type. At the same time, the position offset of the record to be read in the data block is determined based on the storage space of the data block in which it is located and the information of the record stored in the data block. Here, the position offset can be used to display the start position of the record in the data block.

As an example, as shown in FIG. 5A, the executing subject has the position information of the Y of the record from the directory of the disk [vlet_id = 110_3, record_type = 126, record_id = 9, block_id = 3, next_block_id = 4]. You can search for that. At this time, it can be calculated that the position offset in block3 of the record Y is 2M-64KB (64KB = 4M-8 × 504KB) based on the storage space of the data block in Vlet_110_3.

In step 303, the length of the read target record is determined based on the type of the read target record, and the data of the length corresponding to the length of the read target record is read, starting from the position indicated by the position offset.

In this embodiment, the executing subject can determine the length of the read target record based on the type of the read target record. Further, the executing subject can read the data having a length corresponding to the length of the record to be read, starting from the position indicated by the position offset determined in step 302. For example, if the type of record Y is 126, its length can be 504KB. At this time, the executing subject can read the data having a length of 504KB into the memory at one time, starting from the position of 2M-64KB in the block 3.

In step 304, the read data is output as a read target record.

In this embodiment, the executing subject can use the read data as a record to be read, and can perform coding analysis on it. As a result, the analyzed data can be output and transmitted to, for example, a terminal (for example, terminals 101, 102, 103 shown in FIG. 1).

In some embodiments, each record may include a field of its stored location information in order to restore the index information if all index information is lost. As shown in FIG. 5B, a Record Guard data structure is used at the last position of the record (eg, 24 fixed bytes).

Optionally, the executor can verify the Record Guard information prior to coding analysis. At the same time, it is possible to detect whether or not the supplemented predetermined data (for example, 0) exists at a predetermined position of the read data. If the predetermined data exists, the predetermined data in the read data can be removed or ignored. Furthermore, the read data after verification that does not include predetermined data is analyzed.

In the method for storing data in this example, the steps of reading the data were increased and the reading process was described in detail. The method for storing data in the present invention has been enhanced and completed. In addition, the read and input processes are all one IOPS (Input / Output Operations Per Second, the number of times of performing read and input (I / O) operations per second). In this way, it helps to expand the scope of the method.

Further, FIG. 4 shows a flow 400 of another embodiment of the method for storing the data according to the present invention. The method for storing the data includes the following steps.

In step 401, the location information of the record to be deleted is searched from the directory.

In this embodiment, the executing subject of the method for storing data (for example, the server 105 shown in FIG. 1) can search the directory for the record to be deleted. When the record to be deleted is searched, the location information can be obtained from the directory.

In step 402, based on the position information of the record to be deleted, the current storage information of the type to which the record to be deleted belongs in the corresponding storage file is acquired.

In this embodiment, the executing subject can determine the storage file in which the record is located based on the location information of the record to be deleted. Then, the current storage information of the type to which the storage file belongs can be acquired.

As an example, as shown in FIG. 5A, the executing subject indicates that the position information of the record Z is [velt_id = 110_3, record_type = 128, record_id = 2, block_id = 37, next_block_id = 0] from the directory of the disk. Can be detected. At this time, the executing subject can acquire the current storage information of type = 128 in Vlet_110_3, that is, block37 (record1-4) and block49 (record5).

In step 403, the last one record can be read from the stored record based on the acquired current storage information, and the read record can be transmitted and stored at the position of the record to be deleted. it can.

In this embodiment, the executor can read the last one record from the current stored record based on the acquired current storage information. In addition, the read record can be transmitted and stored at the position of the record to be deleted.

By way of example, as shown in FIG. 5A, the executing subject can read the record X with record_id = 5. As a result, the record X can be moved and input to the position of record_id = 2. Since the data lengths of the two records are the same, overwriting is possible.

In step 404, the data in the data block before the transfer of the read record is cleared, and the position information of the read record is corrected.

In this embodiment, the executing subject can clear the data (for example, record 5) in the data block (for example, block_id = 49) before the transfer of the read record (for example, record X). In addition, it is necessary to correct the position information of the read record. For example, the corrected position information of the record X is [record_type = 128, record_id = 2, block_id = 37, next_block_id = 0]. The point to be explained is that after deleting the record to be deleted, the related information (for example, the directory of the disk, the current storage information under the corresponding type, the Record Guard of record X, etc.) needs to be further updated.

In step 405, it is determined whether or not data exists in the data block located before the transfer of the read record.

In this embodiment, the executing subject can further determine whether or not the data exists in the data block located before the read record is transferred. For example, as shown in FIG. 5A, after the record X is transferred and stored, the executing subject determines whether or not other data is stored in the block49.

If it is determined in step 406 that no data is stored in the data block, the data block is reclaimed for reassignment.

In this embodiment, if it is determined that no data is stored in the data block, the executing entity can recover the data block for reassignment. For example, the already allocated state of the data block can be modified to the unallocated state. For example, as can be seen from FIG. 5A, after transferring and storing record X, the storage space of block49 is completely empty. At this time, the Block Manager can collect the data block (FreeBlock49).

In the method for storing data in this embodiment, the steps for deleting data have been increased, and the deletion process has been described in detail. The method for storing data in the present invention has been further enhanced and completed. Further, the deletion process corresponds to adding one writing process to one reading process, which is useful for improving the overall performance of the disk. In addition, the continuity of the remaining data storage is ensured so that space fragments do not occur.

It is understandable that in the disks in each of the above embodiments, the size of each storage file constructed therein can be the same, which is advantageous for easy management and improving the efficiency of data processing. The size of each storage file can be different, which helps to meet the demands of different users. Note that the size of the data blocks in different storage files can be the same and can be different. For convenience of management, a configuration file can be installed in each storage file. Here, the configuration file can be used to describe the configuration parameter information of the storage file.

As shown in FIG. 5A, there may be one 512KB file (ie, VletInfo) at the end of the storage file. The file is a simple and efficient storage format for structured data, which can store its own configuration parameter information and can use Protocol (Google Protocol Buffer). It is extensible, easy to use, and fast to analyze, regardless of platform and language. The point to explain is that the content in this file is generated only when the storage file is configured and cannot be modified later.

Here, the configuration parameter information includes a storage file size (for example, 32 GB), a data block size (for example, 2 M), a minimum type in a record (for example, 4 kB), and a maximum type in a record (for example, 512 KB). It can include, but is not limited to, at least one of the intervals between record types (eg, 4 kB). In other words, in addition to storing vletinfo in the last page of the storage file, all other parts can be data blocks of the same size.

It should be noted that the storage of a large number of small files is always a difficult problem to be solved by a distributed storage system. The user generally had to make compromises in terms of read performance, space utilization, deletion efficiency, etc., but in the method of storing data in each embodiment of the present invention, data writing is performed on the disk 1. It can be equivalent to one random read / write. Also, deleting data can be equivalent to adding a write to a single read. In this way, it helps to ensure the read / write performance and deletion efficiency of the disk. In addition, no space fragments are generated when writing or deleting data. In addition, empty storage space can be recovered immediately. In this way, it helps to improve the space utilization of the disk. In other words, the method in the present invention can balance the above requirements of the user, reach a good processing effect during actual use, and can be advantageous in improving the user experience. ..

In the method for storing data in the embodiments of the present invention, all data is read and written in one IOPS unless it spans two data blocks. Here, measures such as adjusting the size of the data block and dividing the data input can reduce or prevent the situation spanning the two data blocks. The read / write performance is basically the same as the consumption time of one random read / write of each disk. In addition, no fragments are generated in the process of writing and deleting, and the space can be recovered immediately. One separate IO write may occur during deletion, but IO consumption is still relatively low compared to conventional solutions.

In addition, there are two main parts in terms of space waste. One is to match the size of the same type of data. The waste of this part depends on the average size of the input data. As can be seen from the statistics, the average input data of the user is generally 256KB. The resulting wasted space is about 0.7%. The other is that the last one data block allocated by each type is not filled. In the worst case, there is only one record with about 127 data blocks. In this case, the waste of space is about (2M × 127) ÷ 32G = 0.7%. Through analysis, it can be seen that all the waste of these two subspaces is acceptable. Therefore, the above method is superior to the existing technology in terms of comprehensive comparison of aspects such as throughput, delay, and space utilization.

It should be explained that the method of the embodiment of the present invention can be used mainly when the scale of data is EB class, the number of reads is larger than that of writes, and the number of writes is larger than that of deletions. This is a very stringent requirement for space utilization (ie, cost savings). At the same time, it requires relatively high read / write delay and throughput.

Subsequently, as shown in FIG. 6, as a realization of the methods shown in each of the above figures, the present invention provides one embodiment of a device for storing data. Examples of the device correspond to examples of the methods shown in each of the above embodiments, and the device can be specifically applied to various electronic devices.

As shown in FIG. 6, the device 600 for storing the data of this embodiment can be configured as a stand-alone storage engine for distributed object storage. Storage files can be configured on a stand-alone storage engine disk. Here, the storage space of the storage file is divided into at least two data blocks, and a linked list structure can be used between the at least two data blocks. The device 600 is configured to search for the current storage information of the type in the storage file and the type determination unit 601 configured to determine the type of the storage target record based on the size of the data of the storage target record. The search unit to be searched, the current storage information includes the information of the currently allocated data block and the currently stored record information of the data blocks, and the records stored under the same type of the storage file. The size of the data is the same, but the size of the data of the records stored under different types is different, the search unit 602 and the remaining storage space in the currently allocated data block based on the current storage information A space determination unit 603 configured to determine if it exists, and in response to the determination that there is no remaining storage space, assign a new data block to the type and store the stored record with new data. A first allocation unit 604 configured to be stored in a block can be provided.

In one selectable embodiment of this embodiment, the device 600 responds to the determination that there is remaining storage space, and is the remaining storage space less than the size of the data in the storage record? A sizing unit (not shown in FIG. 6) configured to further determine whether or not, and storage in response to the determination that the remaining storage space is smaller than the size of the data in the record to be stored. A second allocation unit configured to store some data in the target record in the remaining storage space, allocate a new data block to the type, and store the remaining data in the storage target record (shown in FIG. 6). It is possible to further prepare for.

Optionally, the storage space of at least two data blocks can be the same, and the data size of records stored in different types can all be an integral multiple of a given number.

Further, the device 600 is configured to update the current storage information of the type, generate the position information after the storage target record is stored, and output the position information (shown in FIG. 6). Can be further provided. Here, the position information includes at least one of a record type, a record identifier, and an identifier of the data block in which the record is located.

In some embodiments, at least two storage files can be configured on the disk, and index information for each storage file can be stored in a directory on the disk.

The device 600 can be selected based on a first position search unit (not shown in FIG. 6) configured to search the position information of the record to be read from the directory and the position information of the record to be read. Read based on the type of record to be read and the type of corresponding storage file, the confirmation unit (not shown in FIG. 6) configured to determine the position offset in the located data block, and the type of record to be read. Determine the length of the target record, read the data whose length is the length of the read target record, starting from the position indicated by the position offset, and output the read data as the read target record. A reading unit (not shown in FIG. 6) that is configured can be further provided.

Further, the device 600 has a second position search unit (not shown in FIG. 6) configured to search the position information of the record to be deleted from the directory, and a deletion target based on the position information of the record to be deleted. Currently stored based on the acquisition unit (not shown in FIG. 6) configured to acquire the current storage information of the type to which the record belongs in the corresponding storage file and the current storage information acquired. A transfer unit (not shown in FIG. 6) configured to read the last record from a record and transfer and store the read record to the position of the record to be deleted, and before the transfer of the read record. A correction unit (not shown in FIG. 6) configured to clear the data in the data block of the above and to correct the position information of the read record can be further provided.

In some embodiments, the device 600 determines if data still exists in the data block located before the read record is transferred and the data is not stored in the data block. , A recovery unit (not shown in FIG. 6) configured to recover the data block for reassignment can be further provided.

In some embodiments, the device 600 is configured to code the sub-objects in at least one sub-object obtained by dividing the storage-waiting object to obtain a copy. A record generation unit (not shown in FIG. 6) configured to generate a storage target record of the sub-object by arranging the unit (not shown in FIG. 6) and the explanatory information and data of the duplicate. ), And can be further prepared.

Selectably, the record generation unit is configured to determine whether the size of the data in the duplicate after arraying is the same as one of the sizes of data corresponding to each type in the storage file. If the size of the data of the confirmed subsystem and the duplicate is different from the size of the data corresponding to each type and smaller than the size of the data corresponding to some types, zero after the data of the duplicate. Fill with and make the size of the data of the duplicate after filling with zero the same as the size of the data corresponding to the target type, which is the smallest type of the corresponding data among the some types, and make the sub-object If the size of the data of the first generation subsystem configured to generate one storage target record of the above and the duplicate is larger than the maximum value of the size of the data corresponding to each type, it is divided into the duplicates and Fill with zeros so that the size of the data in each duplicate after being split is the same as one of the sizes of data corresponding to each type, and generate at least two storage records for that sub-object. A second generation subsystem, which is configured, can be provided.

It is understandable that some of the units described in the device 600 correspond to each step in the method described in FIGS. Thereby, the operations, functions, and beneficial effects produced that describe the above method are similarly applied to the device 600 and the units contained therein, which will not be repeated herein.

With reference to FIG. 7, a timing diagram of an embodiment of a system for storing data according to the present invention is shown.

The system for storing data in this embodiment can include a first subsystem, a second subsystem, and a third subsystem in which the stand-alone storage engine described in each of the above embodiments is installed. The first subsystem receives the storage request including the storage waiting object sent by the user, divides the storage waiting object into at least one sub-object, and the correspondence between the storage-waiting object and at least one sub-object. Is configured to be transmitted to the second subsystem and at least one sub-object to be transmitted to the third subsystem. The second subsystem is configured to store in a list the correspondence between the storage-waiting object and at least one sub-object. The third subsystem is configured to perform coding and arranging processing on the sub-objects in at least one sub-object to generate a storage target record of the sub-object and store the generated storage target record. ..

As shown in FIG. 7, in step 701, the first subsystem can receive a storage request including a storage waiting object transmitted by the user.

In this embodiment, the first subsystem can receive a storage request including a storage waiting object transmitted by the user by a wired connection method or a wireless connection method. Here, the storage-waiting object can be data because it is a distributed object storage system.

In step 702, the first subsystem can divide the storage-waiting object into at least one sub-object. Here, the division method can be set according to the actual demand.

In step 703, the first subsystem can transmit the correspondence between the storage-waiting object and at least one subsystem to the second subsystem, and at least one subsystem to the third subsystem. be able to.

In step 704, the second subsystem can store in the list the correspondence between the storage-waiting object and at least one sub-object.

In step 705, the third subsystem performs coding and arranging processing on the sub-objects in at least one sub-object to generate a storage target record of the sub-object, and stores the generated storage target record. can do. This can be referred to in the relevant description of the embodiment of FIG. 2 and will not be repeated herein.

In certain selectable embodiments of this embodiment, for example, in step 706, the third subsystem can send response information to the first subsystem to indicate the completion of data storage.

Then, in step 707, when the first subsystem receives the response information, it generates a query identifier of the storage waiting object and feeds back the query identifier to the user. In this way, the user can access the object data it represents by the query identifier. Here, the query identifier includes, but is not limited to, at least one character code such as a number, an alphabet, or a character.

Optionally, for example, in step 708, the first subsystem can receive a read request containing the query identifier sent by the user.

Subsequently, in step 709, the first subsystem can send the query identifier in the read request to the second subsystem.

Then, in step 710, the second subsystem can acquire the sub-object list corresponding to the object indicated by the query identifier in the read request, and can send the sub-object list to the third subsystem. ..

Then, in step 711, the third subsystem may read the corresponding record according to the sub-object list, parse the read record to obtain the object data, and send the object data to the first subsystem. it can. This can be referred to in the relevant description of the embodiment of FIG. 3 and will not be repeated herein.

Finally, in step 712, the first subsystem can feed back the object data to the user.

In one application scenario, the first subsystem may further receive a delete request containing the query identifier sent by the user. In this way, the third subsystem can delete the object data indicated by the deletion request, according to the description related to the embodiment of FIG. I will not repeat it here.

It is understandable that the first subsystem, the second subsystem and the third subsystem can be located in different electronic devices (for example, three servers) and are the same electronic device (shown in FIG. 1). It can also be located on the server 105). Moreover, in some embodiments, when the first subsystem is provided with the function of the second subsystem, the system in this embodiment does not need to install the second subsystem.

In the system for storing the data of this embodiment, a new data storage method is applied, that is, the data is classified and stored according to the size of the data. In this way, not only can distributed storage of data be realized, but also data security is improved. At the same time, the overall data processing performance can be improved and the operating cost can be reduced.

Hereinafter, with reference to FIG. 8, a schematic structural diagram of a computer system 800 applied to realize an electronic device (for example, the server 105 shown in FIG. 1) according to an embodiment of the present application is shown. The electronic device shown in FIG. 8 shows only one embodiment and does not limit the function and range of use of the embodiment of the present invention.

As shown in FIG. 8, the computer system 800 performs various appropriate operations and processes by a program in the read-only memory (ROM) 802 or a program loaded from the storage portion 808 into the random access memory (RAM) 803. It includes a central processing unit (CPU) 801 that can be executed. The RAM 803 further stores various programs and data necessary for operating the system 800. The CPU 801 and the ROM 802 and the RAM 803 are interconnected via the bus 804. The input / output (I / O) interface 805 is also connected to the bus 804.

Input part 806 with keyboard, mouse, etc., cathode ray tube (CRT), output part 807 with liquid crystal display (LCD), speaker, etc., storage part 808 with hard disk, LAN card, modem, etc. The communication portion 809 of the network interface card including the above is connected to the I / O interface 805. The communication portion 809 executes communication processing via a network such as the Internet. The driver 810 is also connected to the I / O interface 805 as needed. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is attached to a driver 810 as needed, and a computer program read from the removable medium 811 is installed in a storage portion 808 as needed.

In particular, in the embodiment according to the present invention, the process described above with reference to the flowchart can be realized as a computer software program. For example, an embodiment of the present disclosure comprises a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising program code that executes the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication portion 809 and / or installed from the removable medium 811. When the computer program is executed by the central processing unit (CPU) 801 it performs the above functions limited to the methods of the present invention. It should be explained that the computer-readable medium described in the present invention can be a computer-readable signal medium, a computer-readable storage medium, or any combination of both. The computer-readable storage medium can be, but is not limited to, for example, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any combination thereof. More specific examples of computer-readable storage media are electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only. It includes, but is not limited to, a memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage element, a magnetic storage element, or any combination thereof. In the present invention, a computer-readable medium can be any tangible medium that comprises or stores a program, which is used by, or combined with, a command execution system, device, or element. Will be done. In the present invention, the computer-readable signal medium carries a computer-readable program code and can be provided for a baseband or for a data signal propagated as part of a carrier wave. The data signal thus propagated can take various forms including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which is used by, or coupled with, a command execution system, device or element. Programs can be transmitted, propagated, or transported. The program code contained on a computer-readable medium can be transmitted by any suitable medium, including, but not limited to, wireless, wire, fiber optic cable, RF, or any suitable combination thereof.

Flow charts and block diagrams in the drawings show the structure, function, and operation of feasible embodiments of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram can represent a module, program segment or part of code, the module, program segment or part of code to achieve a defined logical function. It comprises one or more executable commands of. Also note that in alternative embodiments, the functions shown in the blocks may occur in a different order than those shown in the drawings. For example, two blocks displayed in succession may actually be executed substantially in parallel, or they may be executed in reverse order depending on the functions included. Also, each block of the block diagram and / or flowchart, and the combination of blocks of the block diagram and / or flowchart, is implemented in a dedicated hardware-based system that performs a specified function or operation, or dedicated hardware. It should also be noted that this can be achieved by a combination of and computer directives.

The units referred to in the embodiments of the present invention can be implemented by software or can also be implemented by hardware. The described unit can also be installed in a processor described as, for example, a processor comprising a type determination unit, a search unit, a space determination unit and a first allocation unit. Here, the names of these units may not limit the unit itself, for example, the type determination unit also "depends on the size of the data of the storage target record, the type of the storage target record. It can also be described as a "determining unit".

In another aspect, the invention further provides a computer-readable medium, which can be included in or separately exists in the electronic device described in the embodiments and is installed in the electronic device. Can not be. The computer-readable medium can carry one or more programs, and when one or more programs are executed by the server, the electronic device depends on the size of the data in the record to be stored. , Determine the type of record to be stored, and search the storage file for the current storage information under the type, which contains information on the currently allocated data block and information on the currently stored record of the data blocks. So, the size of the data of the records stored under the same type of the storage file is the same, the size of the data of the records stored under the different type is different, and it is currently allocated according to the current storage information. Determine if there is remaining storage space in the given data block, and in response to the determination that there is no remaining storage space, assign a new data block under the type to store the records to be stored. Store in a new data block.

The above description is merely a description of preferred embodiments of the present invention and principles of applied techniques. The scope of the present invention referred to in the present invention is not limited to the specific combination of the above technical features, and at the same time, the above technical features or equivalent features without departing from the above concept of the invention. It was formed in any combination. For example, other technical solutions formed by exchanging the above-mentioned features and technical features having functions similar to those disclosed in the present invention but not limited to them should be covered. Those skilled in the art should understand.

Claims (17)

A method for storing data, the method being used for a stand-alone storage engine of distributed object storage, a storage file is configured on the disk of the stand-alone storage engine, and the storage space of the storage file is at least two pieces of data. It is divided into blocks and uses a link list structure between the at least two data blocks.
The method is
Determining the type of the storage target record based on the size of the data of the storage target record,
Searching for the current storage information of the type in the storage file, the current storage information includes information on the currently allocated data block and information on the currently stored record of the data block. The size of the data of the records stored in the same type of the storage file is the same, and the size of the data of the records stored in different types of the storage file is different.
Determining whether or not there is remaining storage space in the currently allocated data block based on the current storage information.
To store data, including allocating a new data block to the type and storing the storage target record in the new data block in response to the determination that there is no remaining storage space. Method. In response to the determination that there is the remaining storage space, it is further determined whether or not the remaining storage space is smaller than the size of the data of the storage target record.
In response to determining that the remaining storage space is smaller than the size of the data in the storage target record, storing some data in the storage target record in the remaining storage space.
The method for storing the data according to claim 1, wherein a new data block is assigned to the type and the remaining data of the storage target record is stored in the new data block, and further includes. .. The data according to claim 1, wherein the storage space of the at least two data blocks is the same, and the size of the data of the records stored in different types is all an integral multiple of a predetermined numerical value. A way to remember. It further comprises updating the current storage information of the type, generating the stored location information of the storage target record, and outputting the location information.
The data according to any one of claims 1 to 3, wherein the position information includes at least one of the type of the record, the identifier of the record, and the identifier of the data block in which the record is located. How to do it. The method for storing the data according to claim 4, wherein at least two storage files are configured on the disk, and index information of each storage file is stored in the directory of the disk. Searching for the location information of the record to be read from the directory
Based on the position information of the read target record, the position offset in the data block located in the type of the read target record and the corresponding storage file is determined.
Based on the type of the read target record, the length of the read target record is determined, and the data of the length corresponding to the length of the read target record is read starting from the position displayed by the position offset. , The method for storing the data according to claim 5, wherein the read data is output as the read target record, and further includes. Searching for the location information of the record to be deleted from the directory
Acquiring the current storage information of the type to which the deletion target record belongs in the corresponding storage file based on the position information of the deletion target record.
Based on the acquired current storage information, the last record is read from the currently stored record, and the read record is transferred to the position of the deletion target record and stored.
The data according to claim 5, wherein the data in the data block before the transfer of the read record is cleared, the position information of the read record is corrected, and the data is further included. The way for. The method for storing the data after clearing the data in the pre-transfer data block of the read record is
Determining whether or not the data exists in the data block where the read record was located before transfer
The method for storing the data according to claim 7, wherein if the data block does not have data, the data block is recovered for reassignment, and further comprises. A method for storing the data before determining the type of the storage target record, based on the size of the data in the storage target record, is
Coding the sub-object to the sub-object in at least one sub-object obtained by dividing the storage waiting object to obtain a duplicate.
The invention according to any one of claims 1 to 8, further comprising performing an arrangement process on the explanatory information and data of the duplicate to generate a storage target record of the sub-object. A method for storing data in. Creating a storage target record for the sub-object
Determining whether the size of the arrayed duplicate data is the same as one of the sizes of data corresponding to each of the types in the storage file.
If the size of the duplicate data is all different from the size of the data corresponding to each of the types and is smaller than the size of some of the data corresponding to the type, the duplicate data is padded with zeros to zero. The size of the data of the duplicate after filling is the same as the size of the data corresponding to the target type, and one storage target record of the sub-object is generated.
When the size of the data of the duplicate is larger than the maximum value of the size of the data corresponding to each of the types, the size of the data of each duplicate after the division is divided into the above types by dividing the duplicate and filling it with zeros. Including making it the same as one of the corresponding data sizes and generating at least two storage target records of the sub-object.
The method for storing the data according to claim 9, wherein the target type is the type having the smallest size of the corresponding data among the partial types. A device for storing data, the device is configured in a stand-alone storage engine of distributed object storage, a storage file is configured in the disk of the stand-alone storage engine, and the storage space of the storage file is at least two data. It is divided into blocks and uses a link list structure between the at least two data blocks.
The device
A type determination unit configured to determine the type of the storage target record based on the size of the data in the storage target record.
A search unit configured to search for the current storage information of the type in the storage file, the current storage information being currently stored in the currently allocated data block information and the data block. The search unit and the search unit, which include record information, have the same size of data in the storage file stored in the same type, but different in size of data in records stored in different types.
A space determination unit configured to determine whether or not there is remaining storage space in the currently allocated data block based on the current storage information.
A first allocation unit configured to allocate a new data block to the type and store the storage target record in the new data block in response to the determination that there is no remaining storage space. A device for storing data. A system for storing data
It comprises a first subsystem, a second subsystem, and a third subsystem in which the stand-alone storage engine according to any one of claims 1 to 10 is installed.
The first subsystem receives a storage request including a storage-waiting object transmitted by a user, divides the storage-waiting object into at least one sub-object, and the storage-waiting object and the at least one sub-object. It is configured to transmit the correspondence between them to the second subsystem and at least one of the sub-objects to the third subsystem.
The second subsystem is configured to store in a list the correspondence between the storage-waiting object and the at least one sub-object.
The third subsystem performs coding and arranging processing on the sub-objects in the at least one sub-object to generate a storage target record of the sub-object, and stores the generated storage target record. A system for storing data that is configured to. The third subsystem is further configured to send response information to indicate the completion of data storage to the first subsystem.
The first subsystem is further configured to generate a query identifier of the storage waiting object when the response information is received and to feed back the query identifier to the user. A system for storing the described data. The first subsystem is further configured to receive a read request including a query identifier sent by the user and send the query identifier in the read request to the second subsystem.
The second subsystem is further configured to acquire a sub-object list corresponding to the object indicated by the query identifier in the read request and send the sub-object list to the third subsystem.
The third subsystem reads the corresponding record based on the sub-object list, analyzes the read record to acquire the object data, and the first subsystem feeds the object data back to the user. The system for storing the data according to claim 13, wherein the object data is further configured to be transmitted to the first subsystem. It ’s an electronic device,
With one or more processors
A storage device for storing one or more programs,
An electronic device in which, when the one or more programs are executed by the one or more processors, the one or more processors realizes the method according to any one of claims 1 to 10. A computer-readable medium that stores computer programs
A computer-readable medium in which the method according to any one of claims 1 to 10 is realized when the computer program is executed by a processor. It ’s a computer program
A computer program that realizes the method according to any one of claims 1 to 10 when the computer program is executed by a processor.