CN111382123A - File storage method, device, equipment and storage medium - Google Patents
File storage method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN111382123A CN111382123A CN201811627718.8A CN201811627718A CN111382123A CN 111382123 A CN111382123 A CN 111382123A CN 201811627718 A CN201811627718 A CN 201811627718A CN 111382123 A CN111382123 A CN 111382123A
- Authority
- CN
- China
- Prior art keywords
- storage
- file
- stored
- block
- blocks
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 41
- 238000004590 computer program Methods 0.000 claims description 10
- 238000013500 data storage Methods 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 3
- 230000003287 optical effect Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 3
- 239000013307 optical fiber Substances 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The embodiment of the invention discloses a file storage method, a file storage device, file storage equipment and a file storage medium. The method comprises the following steps: creating a plurality of storage blocks with set sizes according to the size of a file to be stored, and generating an index file of the file to be stored according to the information of the plurality of storage blocks; scanning storage block information in the index file according to storage requests of a plurality of clients, and determining a plurality of target blocks meeting storage conditions; respectively storing the data with set size in the file to be stored into a target block; and determining that the target blocks are completely stored, returning and executing the operation of scanning the storage block information in the index file according to the storage request of at least one client, and determining at least one target block meeting the storage condition until all the storage blocks are determined to be completely stored. The file storage method provided by the embodiment of the invention can write the data of the file to be stored into a plurality of storage blocks simultaneously, improve the storage rate of the large file and save time.
Description
Technical Field
The embodiment of the invention relates to the technical field of data storage, in particular to a file storage method, a file storage device, file storage equipment and a storage medium.
Background
In internet products, with the increasing popularity of multimedia applications and big data applications, very large single files, such as files of several hundred GB and even TB level, are increasing. Storage of a single large file at the level of several hundred GB, and even at the level of TB, presents corresponding challenges to current distributed storage technologies.
The current Distributed storage system, such as a Hadoop Distributed File System (HDFS), adopts the following scheme: the method comprises the steps of dividing a super-large file into default 64MB data blocks, respectively storing the data blocks on different data disks, and recording which data blocks the current file consists of in a file manager (NameNode). In the process of writing in the large file, the client writes data into each storage area block in sequence to form a data block. The above scheme has the following disadvantages: the current file can only be written into each data block in sequence, which affects the data storage rate and wastes time.
Disclosure of Invention
The embodiment of the invention provides a file storage method, a file storage device, file storage equipment and a file storage medium, which are used for realizing parallel storage of large files, improving the storage rate of the large files and saving time.
In a first aspect, an embodiment of the present invention provides a file storage method, which is characterized by including:
creating a plurality of storage blocks with set sizes according to the size of a file to be stored, and generating an index file of the file to be stored according to the information of the plurality of storage blocks;
scanning storage block information in the index file according to a storage request of at least one client, and determining at least one target block meeting storage conditions;
respectively storing the data with set size in the file to be stored into a target block;
and determining that the target blocks are completely stored, returning and executing the operation of scanning the storage block information in the index file according to the storage request of at least one client, and determining at least one target block meeting the storage condition until all the storage blocks are determined to be completely stored.
Further, the information of the memory block includes: the number of the storage block, the current state of the storage block, the client identification information of the data which is written into the storage block recently, the time of writing the data into the storage block recently, and the position of the storage block.
Further, creating a plurality of storage blocks with set sizes according to the size of the file to be stored, and generating an index file of the file to be stored according to the information of the plurality of storage blocks, including:
receiving a creation request sent by a client, wherein the creation request comprises the size and the hash value of a file to be stored;
inquiring whether an index file of the file to be stored is stored in a key value database according to the size and the hash value of the file to be stored;
if not, determining a storage area according to the size of the file to be stored;
dividing the storage area into a plurality of storage blocks with set sizes, and acquiring information of the plurality of storage blocks;
and generating an index file of the file to be stored according to the information of the plurality of storage blocks.
Further, after generating the index file of the file to be stored according to the information of the plurality of storage blocks, the method further includes:
and storing the size, the hash value and the index file of the file to be stored into a key value database.
Further, the storage conditions include: the current state of the memory block is initialized or in uploading, the client identification information of the data which is written into the memory block recently is empty, and the time of writing the data into the memory block recently does not exceed the set time from the current time.
Further, determining that the target block is completely stored includes:
judging whether the client identification information of the target block, which has recently written data into the storage block, is the same as the identification information of the current client,
if the data are the same, acquiring a first check value sent by the current client, wherein the first check value is obtained by the client through calculation according to the data written into the target block;
calculating a second check value according to the currently stored data of the target block;
and if the first check value is the same as the second check value, the target block finishes storing.
Further, after determining that the target block is completely stored, the method further includes:
the current state of the target block is updated to be stored.
In a second aspect, an embodiment of the present invention further provides a file storage apparatus, including:
the storage block creating module is used for creating a plurality of storage blocks with set sizes according to the size of the file to be stored and generating an index file of the file to be stored according to the information of the plurality of storage blocks;
the target block determining module is used for scanning the storage block information in the index file according to the storage request of at least one client and determining at least one target block meeting the storage condition;
the data storage module is used for respectively storing the data with set size in the file to be stored into the target blocks;
and the storage completion determining module is used for determining that the target blocks are completely stored, returning and executing the operation of scanning the storage block information in the index file according to the storage request of at least one client and determining at least one target block meeting the storage condition until all the storage blocks are determined to be completely stored.
In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the file storage method according to the embodiment of the present invention when executing the computer program.
In a fourth aspect, the embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the file storage method according to the embodiment of the present invention.
The method comprises the steps of firstly creating a plurality of storage blocks with set sizes according to the size of a file to be stored, generating an index file of the file to be stored according to information of the storage blocks, then scanning storage block information in the index file according to a storage request of at least one client, determining at least one target block meeting storage conditions, then respectively storing data with the set sizes in the file to be stored into the target blocks, finally determining that the target blocks finish storage, returning to execute the operation of scanning the storage block information in the index file according to the storage request of at least one client, and determining at least one target block meeting the storage conditions until all the storage blocks finish storage. According to the file storage method provided by the embodiment of the invention, at least one target block meeting the storage condition is determined according to the storage block information, the data with the set size is stored in the target block, the data of the file to be stored can be written into a plurality of storage blocks simultaneously, the storage rate of the large file is improved, and the time is saved.
Drawings
FIG. 1 is a flowchart of a file storage method according to a first embodiment of the present invention;
FIG. 2 is a schematic structural diagram of a file storage apparatus according to a second embodiment of the present invention;
fig. 3 is a schematic structural diagram of a computer device in a third embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Example one
Fig. 1 is a flowchart of a file storage method according to an embodiment of the present invention, where this embodiment is applicable to a case of storing a large file, and the method may be executed by a file storage device, where the file storage device may be composed of hardware and/or software, and may be generally integrated in a device with a file storage function, where the device may be an electronic device such as a server, a mobile terminal, or a server cluster. As shown in fig. 1, the method specifically includes the following steps:
step 110, creating a plurality of storage blocks with set sizes according to the size of the file to be stored, and generating an index file of the file to be stored according to the information of the plurality of storage blocks.
The set size may be 64MB, and if the size of the file to be stored is 100G, 1600 blocks need to be created. The information of the memory block may include the number of the memory block, the current state of the memory block, the client identification information of the most recent data written to the memory block, the time of the most recent data written to the memory block, and the location of the memory block. The current state of the block may be initialized, uploaded, or stored, and the location of the block refers to the location of the access block in the storage disk. The index file is used for storing the information of the created storage block. In this embodiment, the index file is stored in the disk and not stored in the intermediate node NameNode, so as to avoid the storage bottleneck of the intermediate node.
Optionally, a plurality of storage blocks with set sizes are created according to the size of the file to be stored, and an index file of the file to be stored is generated according to the information of the plurality of storage blocks, which may be implemented in the following manner: receiving a creation request sent by a client, wherein the creation request comprises the size and the hash value of a file to be stored; inquiring whether an index file of a file to be stored is stored in a key value database or not according to the size and the hash value; if not, determining a storage area according to the size of the file to be stored; dividing a storage area into a plurality of storage blocks with set sizes, and acquiring information of the plurality of storage blocks; and generating an index file of the file to be stored according to the information of the plurality of storage blocks.
The size of the file to be stored and the hash value form a key value (key value), and the form may be: the file-sha1| file-size, wherein the file-sha1 is the hash value of the file to be stored, and the file-size is the size of the file to be stored. The key value database is used for storing key values. Table 1 shows a representation of the key values in the present embodiment.
TABLE 1
In table 1, sha1| size is a combination of file _ sha1 and file _ size of a file, and value refers to an index file of a file to be stored.
Specifically, the NameNode firstly allocates a storage address (file-id) for a file to be stored, sends the file-id to a client, calls a creation interface (createFile) according to the file-id, transmits the file _ sha1 and the file _ size through the creation interface, inquires whether an index file of the file to be stored is stored in a key value database according to the file _ sha1 and the file _ size, returns the address of the index file to the client if the index file is stored, determines a storage area according to the file _ size if the index file is not stored, divides the storage area into a plurality of storage blocks with the size of 64MB, acquires information of each storage block, and generates the index file of the file to be stored according to the information of each storage block. Illustratively, the format of the index file is as follows:
#EXTBIGOLFILE
#EXT-X-VERSION:1
#EXT-X-HASH:ee0ff1eda631c39c918ad81088628b84
#EXT-X-SIZE:107374182400
#EXT-X-PART-SIZE:67108864
#EXTBLOCK:0,
#STATUS:2
#UPUID:12345
#UPTIME:1531815189
g1/M07/01/A7/x1tIGltNjfqIAz-fAAEAE-rO9FgAAClWgDdCkIAAQAr902.part
#EXTBLOCK:1,
#STATUS:2
#UPUID:12345
#UPTIME:1531815189
g1/M07/01/A7/x1tIGltNjfqIAz-fAAEAE-rO9FgAAClWgDdCkIAAQAr802.part
……
#EXTBLOCK:1600,
#STATUS:2
#UPUID:12345
#UPTIME:1531815189
g1/M07/01/A7/x1tIGltNjfqIAz-fAAEAE-rO9FgAAClWgDdCkIAAQAr702.pa
rt
#EXT-X-ENDLIST
table 2 shows the meaning of each field of the index file in this embodiment.
TABLE 2
Step 120, scanning the storage block information in the index file according to the storage request of at least one client, and determining at least one target block meeting the storage condition.
Wherein the storage conditions include: the current state of the memory block is initialized or in uploading, the client identification information of the data which is written into the memory block recently is empty, and the time of writing the data into the memory block recently does not exceed the set time from the current time. In this embodiment, the memory blocks satisfying the above 3 conditions are determined as the target blocks. The set time may be set to any value between 30-50 seconds.
In this embodiment, the files are stored by a plurality of parallel clients, and after the current client completes the storage of one storage block, the data continues to be stored in the next target block. Specifically, the client calls a query file state interface (statFile) and transmits a file-id parameter, so that the disk scans storage block information in the index file according to the file-id parameter to acquire a target block meeting storage conditions.
Exemplarily, after the client inputs the file-id parameter through the query file state interface, the returned parameters include: int (STATUS _ OK (0) indicates success, others indicate failure); status (status of current storage block: 1 indicates in upload, 2 indicates storage is complete); up _ block _ index (index information of the current storage block needs to be uploaded, if up _ block _ index is-1 and status is in uploading, it means that the current storage block is being uploaded by other clients and the current client does not need to upload); block _ size (the size of the current memory block into which data is not stored).
Step 130, storing the data with the set size in the file to be stored into the target blocks respectively.
Specifically, after the target parameter is determined, the client calls a write interface (writeFile), file _ id of the file, client identification information (uid), a sequence number (block _ index) of the target block, a position (block _ offset) where data already stored in the target block arrives, and data (buffer) to be stored are transmitted through the write interface, and the disk stores the data to be stored in the target block according to the transmitted parameter.
Optionally, after the client end transmits the parameters through the write-in interface, the disk needs to determine the state of the target block again, determine whether the state meets the storage condition, if so, write the data to be stored into the target block, and return a STATUS _ OK (0) parameter to the client end, indicating that the write-in is successful; if not, returning a parameter of STATUS _ CONFLICT _ OFFSET (-3) to the client, wherein the parameter indicates that the current storage block is being written by other clients and the target block needs to be acquired again.
Step 140, determining that the target block is completely stored, and determining that all the storage blocks are completely stored, if not, returning to step 130, and if all the storage blocks are completely stored, executing step 150.
Specifically, determining that the target block completes storage may be implemented in the following manner: judging whether the identification information of the client, which has recently written data into the storage block, of the target block is the same as the identification information of the current client, if so, acquiring a first check value sent by the current client, wherein the first check value is calculated and acquired by the client according to the data written into the target block; calculating a second check value according to the currently stored data of the target block; and if the first check value is the same as the second check value, the target block finishes storing.
Wherein, the check value can be crc-32 information. Specifically, the client calls a verification interface (verifyBlock), and transmits the following parameters through the verification interface: file _ id of the file, client identification information (uid), sequence number of the target block (block _ index), and a first check value. And the disk judges whether the client identification information of the target block, which is used for writing data into the storage block recently, is the same as the identification information of the current client according to the transmitted parameters, if the client identification information of the target block is not the same as the identification information of the current client, the target block is written with data by other clients, and a STATUS _ CONFLICT _ OFFSET (-3) parameter is returned to the client. If the first check value and the second check value are the same, the target block finishes storing, a STATUS _ OK (0) parameter is returned to the client to indicate that the storing is finished, and if the first check value and the second check value are different, a STATUS _ VERIFY _ FAILED (-4) parameter is returned to the client to indicate that the first check value and the second check value are not matched.
Optionally, after determining that the target block is completely stored, the method further includes the following steps: the current state of the target block is updated to be stored.
In this embodiment, after the current target block is stored, the client recalls the query file state interface to obtain the target block, so as to continue to write data into the next target block until all the created storage blocks are stored.
Step 150, storage is completed.
According to the technical scheme of the embodiment, a plurality of storage blocks with set sizes are created according to the size of a file to be stored, an index file of the file to be stored is generated according to information of the storage blocks, then storage block information in the index file is scanned according to a storage request of at least one client, at least one target block meeting storage conditions is determined, then data with the set sizes in the file to be stored are respectively stored in the target blocks, the target blocks are determined to be stored, the storage block information in the index file is scanned according to the storage request of the at least one client, and operation of the at least one target block meeting the storage conditions is determined until all the storage blocks are determined to be stored. According to the file storage method provided by the embodiment of the invention, at least one target block meeting the storage condition is determined according to the storage block information, the data with the set size is stored in the target block, the data of the file to be stored can be written into a plurality of storage blocks simultaneously, the storage rate of the large file is improved, and the time is saved.
Example two
Fig. 2 is a schematic structural diagram of a file storage device according to a second embodiment of the present invention. As shown in fig. 2, the apparatus includes: a storage block creation module 210, a target block determination module 220, a data storage module 230, and a storage completion determination module 240.
A storage block creating module 210, configured to create a plurality of storage blocks with set sizes according to the size of a file to be stored, and generate an index file of the file to be stored according to information of the plurality of storage blocks;
a target block determining module 220, configured to scan storage block information in the index file according to a storage request of at least one client, and determine at least one target block that meets a storage condition;
the data storage module 230 is configured to store data with a set size in the file to be stored into target blocks respectively;
and the storage completion determining module 240 is configured to determine that the target block completes storage, return to perform operations of scanning the storage block information in the index file according to the storage request of at least one client, and determine at least one target block meeting the storage condition until it is determined that all the storage blocks complete storage.
Optionally, the information of the storage block includes: the number of the storage block, the current state of the storage block, the client identification information of the data which is written into the storage block recently, the time of writing the data into the storage block recently, and the position of the storage block.
Optionally, the storage block creating module 210 is further configured to:
receiving a creation request sent by a client, wherein the creation request comprises the size and the hash value of a file to be stored;
inquiring whether an index file of the file to be stored is stored in a key value database according to the size and the hash value of the file to be stored;
if not, determining a storage area according to the size of the file to be stored;
dividing the storage area into a plurality of storage blocks with set sizes, and acquiring information of the plurality of storage blocks;
and generating an index file of the file to be stored according to the information of the plurality of storage blocks.
Optionally, the method further includes:
and the key value database storage module is used for storing the size, the hash value and the index file of the file to be stored into the key value database.
Optionally, the storage condition includes: the current state of the memory block is initialized or in uploading, the client identification information of the data which is written into the memory block recently is empty, and the time of writing the data into the memory block recently does not exceed the set time from the current time.
Optionally, the storage completion determining module 240 is further configured to:
judging whether the client identification information of the target block, which has recently written data into the storage block, is the same as the identification information of the current client,
if the data are the same, acquiring a first check value sent by the current client, wherein the first check value is obtained by the client through calculation according to the data written into the target block;
calculating a second check value according to the currently stored data of the target block;
and if the first check value is the same as the second check value, the target block finishes storing.
Optionally, the method further includes:
and the state updating module is used for updating the current state of the target block into the finished storage.
The device can execute the methods provided by all the embodiments of the invention, and has corresponding functional modules and beneficial effects for executing the methods. For details not described in detail in this embodiment, reference may be made to the methods provided in all the foregoing embodiments of the present invention.
EXAMPLE III
Fig. 3 is a schematic structural diagram of a computer device according to a third embodiment of the present invention. FIG. 3 illustrates a block diagram of a computer device 312 suitable for use in implementing embodiments of the present invention. The computer device 312 shown in FIG. 3 is only an example and should not bring any limitations to the functionality or scope of use of embodiments of the present invention. Device 312 is typically a computing device that assumes file storage functionality.
As shown in FIG. 3, computer device 312 is in the form of a general purpose computing device. The components of computer device 312 may include, but are not limited to: one or more processors 316, a storage device 328, and a bus 318 that couples the various system components including the storage device 328 and the processors 316.
The computer device 312 may also communicate with one or more external devices 314 (e.g., keyboard, pointing device, camera, display 324, etc.), with one or more devices that enable a user to interact with the computer device 312, and/or with any devices (e.g., network card, modem, etc.) that enable the computer device 312 to communicate with one or more other computing devices. Such communication may occur via input/output (I/O) interfaces 322. Also, computer device 312 may communicate with one or more networks (e.g., a Local Area Network (LAN), Wide Area Network (WAN), and/or a public Network, such as the internet) via Network adapter 320. As shown, network adapter 320 communicates with the other modules of computer device 312 via bus 318. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the computer device 312, including but not limited to: microcode, device drivers, Redundant processing units, external disk drive Arrays, disk array (RAID) systems, tape drives, and data backup storage systems, among others.
The processor 316 executes various functional applications and data processing, for example, implementing the file storage method provided by the above-described embodiment of the present invention, by executing programs stored in the storage 328.
Example four
The sixth embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the file storage method provided in the sixth embodiment of the present invention.
Of course, the computer program stored on the computer-readable storage medium provided by the embodiment of the present invention is not limited to the method operations described above, and may also perform related operations in the file storage method provided by any embodiment of the present invention.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A file storage method, comprising:
creating a plurality of storage blocks with set sizes according to the size of a file to be stored, and generating an index file of the file to be stored according to the information of the plurality of storage blocks;
scanning storage block information in the index file according to a storage request of at least one client, and determining at least one target block meeting storage conditions;
respectively storing the data with set size in the file to be stored into a target block;
and determining that the target blocks are completely stored, returning and executing the operation of scanning the storage block information in the index file according to the storage request of at least one client, and determining at least one target block meeting the storage condition until all the storage blocks are determined to be completely stored.
2. The method of claim 1, wherein storing information of the block comprises: the number of the storage block, the current state of the storage block, the client identification information of the data which is written into the storage block recently, the time of writing the data into the storage block recently, and the position of the storage block.
3. The method of claim 2, wherein creating a plurality of storage blocks with set sizes according to the size of the file to be stored, and generating an index file of the file to be stored according to information of the plurality of storage blocks comprises:
receiving a creation request sent by a client, wherein the creation request comprises the size and the hash value of a file to be stored;
inquiring whether an index file of the file to be stored is stored in a key value database according to the size and the hash value of the file to be stored;
if not, determining a storage area according to the size of the file to be stored;
dividing the storage area into a plurality of storage blocks with set sizes, and acquiring information of the plurality of storage blocks;
and generating an index file of the file to be stored according to the information of the plurality of storage blocks.
4. The method according to claim 3, further comprising, after generating the index file of the file to be stored according to the information of the plurality of storage blocks:
and storing the size, the hash value and the index file of the file to be stored into a key value database.
5. The method of claim 2, wherein the storage condition comprises: the current state of the memory block is initialized or in uploading, the client identification information of the data which is written into the memory block recently is empty, and the time of writing the data into the memory block recently does not exceed the set time from the current time.
6. The method of claim 2, wherein determining that the target block completes storage comprises:
judging whether the client identification information of the target block, which has recently written data into the storage block, is the same as the identification information of the current client,
if the data are the same, acquiring a first check value sent by the current client, wherein the first check value is obtained by the client through calculation according to the data written into the target block;
calculating a second check value according to the currently stored data of the target block;
and if the first check value is the same as the second check value, the target block finishes storing.
7. The method of claim 1, after determining that the target block is completely stored, further comprising:
the current state of the target block is updated to be stored.
8. A file storage device, comprising:
the storage block creating module is used for creating a plurality of storage blocks with set sizes according to the size of the file to be stored and generating an index file of the file to be stored according to the information of the plurality of storage blocks;
the target block determining module is used for scanning the storage block information in the index file according to the storage request of at least one client and determining at least one target block meeting the storage condition;
the data storage module is used for respectively storing the data with set size in the file to be stored into the target blocks;
and the storage completion determining module is used for determining that the target blocks are completely stored, returning and executing the operation of scanning the storage block information in the index file according to the storage request of at least one client and determining at least one target block meeting the storage condition until all the storage blocks are determined to be completely stored.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1-7 when executing the program.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811627718.8A CN111382123B (en) | 2018-12-28 | 2018-12-28 | File storage method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811627718.8A CN111382123B (en) | 2018-12-28 | 2018-12-28 | File storage method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111382123A true CN111382123A (en) | 2020-07-07 |
| CN111382123B CN111382123B (en) | 2023-06-16 |
Family
ID=71218117
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811627718.8A Active CN111382123B (en) | 2018-12-28 | 2018-12-28 | File storage method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111382123B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111949710A (en) * | 2020-08-17 | 2020-11-17 | 北京锐安科技有限公司 | Data storage method, device, server and storage medium |
| CN113094374A (en) * | 2021-04-27 | 2021-07-09 | 广州炒米信息科技有限公司 | Distributed storage and retrieval method and device and computer equipment |
| CN113468128A (en) * | 2021-07-21 | 2021-10-01 | 上海浦东发展银行股份有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN113704176A (en) * | 2021-07-09 | 2021-11-26 | 奇安信科技集团股份有限公司 | File scanning method, file scanning device, electronic equipment, program product and storage medium |
| CN114237508A (en) * | 2021-12-16 | 2022-03-25 | 中国农业银行股份有限公司 | Data storage method, device, equipment and storage medium |
| CN114327946A (en) * | 2021-12-24 | 2022-04-12 | 北京百度网讯科技有限公司 | Shared memory access control method and device, electronic equipment and automatic driving vehicle |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150161161A1 (en) * | 2013-12-05 | 2015-06-11 | International Business Machines Corporation | Index Writing in a Linear Tape File System |
| CN107728940A (en) * | 2017-09-27 | 2018-02-23 | 柏科数据技术（深圳）股份有限公司 | The method, apparatus and storage medium of a kind of management copy in data-storage system |
| CN107807989A (en) * | 2017-11-03 | 2018-03-16 | 小草数语（北京）科技有限公司 | Small documents processing method and processing device |
| CN109086388A (en) * | 2018-07-26 | 2018-12-25 | 百度在线网络技术（北京）有限公司 | Block chain date storage method, device, equipment and medium |
-
2018
- 2018-12-28 CN CN201811627718.8A patent/CN111382123B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150161161A1 (en) * | 2013-12-05 | 2015-06-11 | International Business Machines Corporation | Index Writing in a Linear Tape File System |
| CN107728940A (en) * | 2017-09-27 | 2018-02-23 | 柏科数据技术（深圳）股份有限公司 | The method, apparatus and storage medium of a kind of management copy in data-storage system |
| CN107807989A (en) * | 2017-11-03 | 2018-03-16 | 小草数语（北京）科技有限公司 | Small documents processing method and processing device |
| CN109086388A (en) * | 2018-07-26 | 2018-12-25 | 百度在线网络技术（北京）有限公司 | Block chain date storage method, device, equipment and medium |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111949710A (en) * | 2020-08-17 | 2020-11-17 | 北京锐安科技有限公司 | Data storage method, device, server and storage medium |
| CN111949710B (en) * | 2020-08-17 | 2024-03-22 | 北京锐安科技有限公司 | Data storage method, device, server and storage medium |
| CN113094374A (en) * | 2021-04-27 | 2021-07-09 | 广州炒米信息科技有限公司 | Distributed storage and retrieval method and device and computer equipment |
| CN113704176A (en) * | 2021-07-09 | 2021-11-26 | 奇安信科技集团股份有限公司 | File scanning method, file scanning device, electronic equipment, program product and storage medium |
| CN113704176B (en) * | 2021-07-09 | 2023-10-31 | 奇安信科技集团股份有限公司 | File scanning method, device, electronic equipment and storage medium |
| CN113468128A (en) * | 2021-07-21 | 2021-10-01 | 上海浦东发展银行股份有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN114237508A (en) * | 2021-12-16 | 2022-03-25 | 中国农业银行股份有限公司 | Data storage method, device, equipment and storage medium |
| CN114327946A (en) * | 2021-12-24 | 2022-04-12 | 北京百度网讯科技有限公司 | Shared memory access control method and device, electronic equipment and automatic driving vehicle |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111382123B (en) | 2023-06-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111382123B (en) | File storage method, device, equipment and storage medium | |
| CN109254733B (en) | Method, device and system for storing data | |
| CN110008045B (en) | Method, device and equipment for aggregating microservices and storage medium | |
| US9785664B2 (en) | Gathering transaction data associated with locally stored data files | |
| US20220253458A1 (en) | Method and device for synchronizing node data | |
| CN109657174B (en) | Method and device for updating data | |
| US11314451B2 (en) | Method and apparatus for storing data | |
| CN108920257B (en) | Transaction processing method, device, equipment and storage medium | |
| CN110019873B (en) | Face data processing method, device and equipment | |
| CN111818145B (en) | File transmission method, device, system, equipment and storage medium | |
| CN111338834B (en) | Data storage method and device | |
| CN110990346A (en) | File data processing method, device, equipment and storage medium based on block chain | |
| CN107862035B (en) | Network reading method and device for conference record, intelligent tablet and storage medium | |
| CN111782614B (en) | Data access method, device, equipment and storage medium | |
| CN111008934B (en) | Scene construction method, device, equipment and storage medium | |
| US9449036B2 (en) | Handling an increase in transactional data without requiring relocation of preexisting data between shards | |
| CN107992457B (en) | Information conversion method, device, terminal equipment and storage medium | |
| CN116069725A (en) | File migration method, device, apparatus, medium and program product | |
| CN112925796A (en) | Write consistency control method, device, equipment and storage medium | |
| CN113810477A (en) | File uploading method, device, equipment and storage medium | |
| US9654140B1 (en) | Multi-dimensional run-length encoding | |
| CN112148705A (en) | Data migration method and device | |
| CN112804312B (en) | File uploading method, equipment and computer readable medium | |
| US11842077B2 (en) | Method, device, and computer program product for transmitting data for object storage | |
| US20210096763A1 (en) | Method, device, and computer program product for managing storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20231008 Address after: 31a, 15 / F, building 30, maple mall, bangrang Road, Brazil, Singapore Patentee after: Baiguoyuan Technology (Singapore) Co.,Ltd. Address before: 511400 floor 23-39, building B-1, Wanda Plaza North, Wanbo business district, 79 Wanbo 2nd Road, Nancun Town, Panyu District, Guangzhou City, Guangdong Province Patentee before: GUANGZHOU BAIGUOYUAN INFORMATION TECHNOLOGY Co.,Ltd. |
|
| TR01 | Transfer of patent right |