CN111382123B - File storage method, device, equipment and storage medium - Google Patents
File storage method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN111382123B CN111382123B CN201811627718.8A CN201811627718A CN111382123B CN 111382123 B CN111382123 B CN 111382123B CN 201811627718 A CN201811627718 A CN 201811627718A CN 111382123 B CN111382123 B CN 111382123B
- Authority
- CN
- China
- Prior art keywords
- storage
- stored
- file
- target block
- block
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 238000004590 computer program Methods 0.000 claims description 7
- 238000013500 data storage Methods 0.000 claims description 7
- 230000003287 optical effect Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 3
- 230000002093 peripheral effect Effects 0.000 description 3
- 238000003491 array Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The embodiment of the invention discloses a file storage method, a device, equipment and a storage medium. The method comprises the following steps: creating a plurality of storage blocks with set sizes according to the sizes of files to be stored, and generating index files of the files to be stored according to the information of the storage blocks; scanning storage block information in the index file according to storage requests of a plurality of clients, and determining a plurality of target blocks meeting storage conditions; respectively storing the data with the set size in the file to be stored into a target block; and determining that the target block is stored completely, returning to execute the operation of scanning the storage block information in the index file according to the storage request of at least one client, and determining at least one target block meeting the storage condition until all the storage blocks are determined to be stored completely. The file storage method provided by the embodiment of the invention can write the data of the file to be stored into a plurality of storage blocks at the same time, thereby improving the storage rate of the large file and saving time.
Description
Technical Field
The embodiment of the invention relates to the technical field of data storage, in particular to a file storage method, device, equipment and storage medium.
Background
In internet products, as multimedia applications and big data applications are becoming more popular, very large single files, such as files of several hundred GB and even TB levels, are becoming more and more popular. The storage of a single large file on the order of hundreds of GB, and even on the order of TB, presents a corresponding challenge to current distributed storage technologies.
Current distributed storage systems, such as Hadoop distributed file systems (Hadoop Distributed File System, HDFS), take the following schemes: the oversized file is split into data blocks of a default 64MB, which are respectively stored on different data discs and which data blocks the current file consists of are recorded in a file manager (NameNode). In the process of writing a large file, the client side sequentially writes data into each storage block to form a data block. The above scheme has the following disadvantages: the current file can only write each data block in turn, which affects the data storage rate and wastes time.
Disclosure of Invention
The embodiment of the invention provides a file storage method, device, equipment and storage medium, which are used for realizing parallel storage of large files, improving the storage rate of the large files and saving time.
In a first aspect, an embodiment of the present invention provides a file storage method, which is characterized in that the method includes:
creating a plurality of storage blocks with set sizes according to the sizes of files to be stored, and generating index files of the files to be stored according to the information of the storage blocks;
scanning storage block information in the index file according to a storage request of at least one client, and determining at least one target block meeting storage conditions;
respectively storing the data with the set size in the file to be stored into a target block;
and determining that the target block is stored completely, returning to execute the operation of scanning the storage block information in the index file according to the storage request of at least one client, and determining at least one target block meeting the storage condition until all the storage blocks are determined to be stored completely.
Further, the information of the memory block includes: the number of the memory block, the current state of the memory block, the client identification information of the most recent data writing to the memory block, the time of the most recent data writing to the memory block, the location of the memory block.
Further, creating a plurality of storage blocks with set sizes according to the sizes of the files to be stored, and generating an index file of the files to be stored according to the information of the storage blocks, including:
receiving a creation request sent by a client, wherein the creation request comprises the size and the hash value of a file to be stored;
inquiring whether an index file of the file to be stored is stored in a key value database according to the size and the hash value of the file to be stored;
if not, determining a storage area according to the size of the file to be stored;
dividing the storage area into a plurality of storage blocks with set sizes, and acquiring information of the storage blocks;
and generating an index file of the file to be stored according to the information of the storage blocks.
Further, after generating the index file of the file to be stored according to the information of the plurality of storage blocks, the method further includes:
and storing the size of the file to be stored, the hash value and the index file into a key value database.
Further, the storage conditions include: in the process of initializing or uploading the current state of the storage block, the identification information of the client terminal which recently writes data into the storage block is empty, and the time for the data to be recently written into the storage block does not exceed the set time from the current time.
Further, determining that the target block completes storing includes:
judging whether the client identification information of the target block which is used for writing data into the storage block recently is the same as the identification information of the current client,
if the data are the same, a first check value sent by the current client is obtained by the client according to the data written into the target block;
calculating a second check value according to the data currently stored in the target block;
and if the first check value is the same as the second check value, the target block is stored.
Further, after determining that the target block is stored, the method further comprises:
and updating the current state of the target block to finish storage.
In a second aspect, an embodiment of the present invention further provides a file storage device, including:
the storage block creating module is used for creating a plurality of storage blocks with set sizes according to the sizes of files to be stored and generating index files of the files to be stored according to the information of the storage blocks;
the target block determining module is used for determining at least one target block meeting the storage condition according to the storage block information in the storage block information scanning index file of the storage request of at least one client;
the data storage module is used for respectively storing the data with the set size in the file to be stored into the target block;
and the storage completion determining module is used for determining that the target blocks complete storage, and returning to execute the operation of scanning the storage block information in the index file according to the storage request of at least one client side and determining at least one target block meeting the storage condition until all the storage blocks are determined to complete storage.
In a third aspect, an embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, where the processor implements the file storage method according to the embodiment of the present invention when executing the program.
In a fourth aspect, embodiments of the present invention further provide a computer readable storage medium having a computer program stored thereon, which when executed by a processor implements a file storage method according to embodiments of the present invention.
According to the embodiment of the invention, a plurality of storage blocks with set sizes are created according to the sizes of files to be stored, an index file of the files to be stored is generated according to the information of the storage blocks, then storage block information in the index file is scanned according to a storage request of at least one client, at least one target block meeting storage conditions is determined, data with set sizes in the files to be stored are respectively stored in the target blocks, finally the target blocks are determined to finish storage, the operation of scanning the storage block information in the index file according to the storage request of at least one client is executed, and at least one target block meeting the storage conditions is determined until all the storage blocks are determined to finish storage. According to the file storage method provided by the embodiment of the invention, at least one target block meeting the storage condition is determined according to the storage block information, and the data with the set size is stored in the target block, so that the data of the file to be stored can be written into a plurality of storage blocks at the same time, the storage rate of a large file is improved, and the time is saved.
Drawings
FIG. 1 is a flow chart of a file storage method according to a first embodiment of the present invention;
FIG. 2 is a schematic diagram of a file storage device according to a second embodiment of the present invention;
fig. 3 is a schematic structural diagram of a computer device in a third embodiment of the present invention.
Detailed Description
The invention is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting thereof. It should be further noted that, for convenience of description, only some, but not all of the structures related to the present invention are shown in the drawings.
Example 1
Fig. 1 is a flowchart of a file storage method according to a first embodiment of the present invention, where the method may be applicable to a case of storing a large file, and the method may be performed by a file storage device, where the device may be composed of hardware and/or software, and may generally be integrated into a device having a file storage function, where the device may be an electronic device such as a server, a mobile terminal, or a server cluster. As shown in fig. 1, the method specifically includes the following steps:
step 110, creating a plurality of storage blocks with set sizes according to the sizes of the files to be stored, and generating an index file of the files to be stored according to the information of the storage blocks.
The set size may be 64MB, and assuming that the size of the file to be stored is 100G, 1600 storage blocks need to be created. The information of the memory block may include the number of the memory block, the current state of the memory block, the client identification information of the last data written to the memory block, the time of the last data written to the memory block, the location of the memory block. The current state of the memory block may be initialized, in the uploading process or after the memory is completed, and the location of the memory block refers to the location of the access block in the memory disk. The index file is used for storing information of the created storage block. In this embodiment, the index file is stored in the disk, and is not stored on the intermediate node NameNode, so as to avoid the situation that the intermediate node has a storage bottleneck.
Optionally, creating a plurality of storage blocks with set sizes according to the sizes of the files to be stored, and generating an index file of the files to be stored according to the information of the plurality of storage blocks may be implemented by the following ways: receiving a creation request sent by a client, wherein the creation request comprises the size and the hash value of a file to be stored; inquiring whether an index file of a file to be stored is stored in a key value database according to the size and the hash value; if not, determining a storage area according to the size of the file to be stored; dividing the storage area into a plurality of storage blocks with set sizes, and acquiring information of the plurality of storage blocks; and generating an index file of the file to be stored according to the information of the plurality of storage blocks.
The size of the file to be stored and the hash value form a key value (key value), which may be: file-sha1|file-size, wherein file-sha1 is the hash value of the file to be stored and file-size is the size of the file to be stored. The key value database is used for storing key values. Table 1 is a representation of key values in this example.
TABLE 1
In table 1, sha1|size is a combination of file_sha1 and file_size of a file, and value refers to an index file of a file to be stored.
Specifically, the NameNode allocates a storage address (file-id) for a file to be stored, sends the file-id to a client, calls a creation interface (createFile) according to the file-id, transmits the file_sha1 and the file_size through the creation interface, inquires whether an index file of the file to be stored is already stored in a key value database according to the file_sha1 and the file_size, returns the address of the index file to the client if the index file is stored, determines a storage area according to the file_size if the index file is not stored, divides the storage area into a plurality of storage blocks with the size of 64MB, acquires information of each storage block, and generates the index file of the file to be stored according to the information of each storage block. Illustratively, the format of the index file is as follows:
#EXTBIGOLFILE
#EXT-X-VERSION:1
#EXT-X-HASH:ee0ff1eda631c39c918ad81088628b84
#EXT-X-SIZE:107374182400
#EXT-X-PART-SIZE:67108864
#EXTBLOCK:0,
#STATUS:2
#UPUID:12345
#UPTIME:1531815189
g1/M07/01/A7/x1tIGltNjfqIAz-fAAEAE-rO9FgAAClWgDdCkIAAQAr902.part
#EXTBLOCK:1,
#STATUS:2
#UPUID:12345
#UPTIME:1531815189
g1/M07/01/A7/x1tIGltNjfqIAz-fAAEAE-rO9FgAAClWgDdCkIAAQAr802.part
……
#EXTBLOCK:1600,
#STATUS:2
#UPUID:12345
#UPTIME:1531815189
g1/M07/01/A7/x1tIGltNjfqIAz-fAAEAE-rO9FgAAClWgDdCkIAAQAr702.pa
rt
#EXT-X-ENDLIST
table 2 shows the meaning of each field of the index file in this embodiment.
TABLE 2
Step 120, determining at least one target block meeting the storage condition according to the storage block information in the storage request scanning index file of at least one client.
Wherein, the storage condition includes: in the process of initializing or uploading the current state of the storage block, the identification information of the client terminal which recently writes data into the storage block is empty, and the time for the data to be recently written into the storage block does not exceed the set time from the current time. In this embodiment, the memory block satisfying the above 3 conditions is determined as the target block. The set time may be set to any value between 30-50 seconds.
In this embodiment, the current client stores the data to the next target block after completing the storage of one storage block through a plurality of parallel clients. Specifically, the client calls a query file state interface (statFile), and a file-id parameter is transmitted, so that the disk scans the storage block information in the index file according to the file-id parameter, and a target block meeting the storage condition is obtained.
Illustratively, after the client inputs the file-id parameter through the query file state interface, the parameters returned are: int (status_ok (0) indicates success, others indicate failure); status (state of current memory block: 1 in upload, 2 in store complete); up_block_index (index information of a current block to be uploaded, if up_block_index= -1 and status is in uploading, it means that the current block is being uploaded by other clients, and the current client does not need to upload); block_size (the size of the current block not storing data).
And 130, respectively storing the data with the set size in the file to be stored into the target blocks.
Specifically, after determining the target parameter, the client calls a write interface (writeFile), and the file_id, the client identification information (uid), the sequence number (block_index) of the target block, the position (block_offset) where the stored data arrives in the target block, and the data (buffer) to be stored of the file are transmitted through the write interface, and the magnetic disk stores the data to be stored in the target block according to the transmitted parameter.
Optionally, after the client side transmits the parameters through the writing interface, the disk needs to judge the state of the target block again, and determine whether the state of the target block meets the storage condition, if so, the data to be stored is written into the target block, and a status_ok (0) parameter is returned to the client side, which indicates that the writing is successful; if not, the parameter of status_connection_offset (-3) is returned to the client, which indicates that the current memory block is being written by other clients and the target block needs to be retrieved.
Step 140, determining that the target block is stored completely, and determining that all the storage blocks are stored completely, if not, returning to step 130, and if all the storage blocks are stored completely, executing step 150.
Specifically, determining that the target block is stored is performed by: judging whether the client identification information of the target block, which is used for writing data into the storage block recently, is the same as the identification information of the current client, if so, acquiring a first check value sent by the current client, wherein the first check value is calculated by the client according to the data written into the target block; calculating a second check value according to the data currently stored in the target block; and if the first check value is the same as the second check value, the target block is stored.
Wherein the check value may be the crc-32 information. Specifically, the client invokes a verification interface (verifyBlock), and passes the following parameters in through the verification interface: file_id of the file, client identification information (uid), sequence number of the target block (block_index), and the first check value. And judging whether the client identification information of the target block, which is used for writing data into the storage block recently, is the same as the identification information of the current client according to the input parameters, if not, indicating that the target block is used for writing data by other clients, and returning a STATUS_CONFLICT_OFFSET (-3) parameter to the client. If the first check value is the same, a first check value transmitted by the client is obtained, a second check value is calculated according to the data stored in the target block currently, if the first check value is the same as the second check value, the target block finishes storing, a STATUS_OK (0) parameter is returned to the client to indicate that storing is finished, and if the first check value is not the same as the second check value, a STATUS_VERIFY_FAILED (-4) parameter is returned to the client to indicate that the first check value is not matched with the second check value.
Optionally, after determining that the target block is stored, the method further includes the following steps: and updating the current state of the target block to finish storage.
In this embodiment, after the current target block completes storage, the client recalls the query file status interface to obtain the target block, so as to continue writing data into the next target block until all the created storage blocks complete storage.
Step 150, storage is completed.
According to the technical scheme, a plurality of storage blocks with set sizes are created according to the sizes of files to be stored, an index file of the files to be stored is generated according to the information of the storage blocks, then storage block information in the index file is scanned according to storage requests of at least one client, at least one target block meeting storage conditions is determined, data with set sizes in the files to be stored are stored in the target blocks respectively, finally the target blocks are determined to finish storage, the operation of scanning the storage block information in the index file according to the storage requests of at least one client is performed, and at least one target block meeting the storage conditions is determined until all the storage blocks are determined to finish storage. According to the file storage method provided by the embodiment of the invention, at least one target block meeting the storage condition is determined according to the storage block information, and the data with the set size is stored in the target block, so that the data of the file to be stored can be written into a plurality of storage blocks at the same time, the storage rate of a large file is improved, and the time is saved.
Example two
Fig. 2 is a schematic structural diagram of a file storage device according to a second embodiment of the present invention. As shown in fig. 2, the apparatus includes: a memory block creation module 210, a target block determination module 220, a data storage module 230, and a storage completion determination module 240.
The storage block creating module 210 is configured to create a plurality of storage blocks with set sizes according to the sizes of files to be stored, and generate an index file of the files to be stored according to the information of the plurality of storage blocks;
a target block determining module 220, configured to determine at least one target block that meets a storage condition according to the storage request of at least one client and scanning the storage block information in the index file;
the data storage module 230 is configured to store the data with the set size in the file to be stored into a target block respectively;
the storage completion determining module 240 is configured to determine that the target block completes storage, and return to perform an operation of scanning the storage block information in the index file according to the storage request of at least one client to determine at least one target block that meets the storage condition until it is determined that all storage blocks complete storage.
Optionally, the information of the storage block includes: the number of the memory block, the current state of the memory block, the client identification information of the most recent data writing to the memory block, the time of the most recent data writing to the memory block, the location of the memory block.
Optionally, the storage block creation module 210 is further configured to:
receiving a creation request sent by a client, wherein the creation request comprises the size and the hash value of a file to be stored;
inquiring whether an index file of the file to be stored is stored in a key value database according to the size and the hash value of the file to be stored;
if not, determining a storage area according to the size of the file to be stored;
dividing the storage area into a plurality of storage blocks with set sizes, and acquiring information of the storage blocks;
and generating an index file of the file to be stored according to the information of the storage blocks.
Optionally, the method further comprises:
and the key value database storage module is used for storing the size of the file to be stored, the hash value and the index file into the key value database.
Optionally, the storage condition includes: in the process of initializing or uploading the current state of the storage block, the identification information of the client terminal which recently writes data into the storage block is empty, and the time for the data to be recently written into the storage block does not exceed the set time from the current time.
Optionally, the storage completion determining module 240 is further configured to:
judging whether the client identification information of the target block which is used for writing data into the storage block recently is the same as the identification information of the current client,
if the data are the same, a first check value sent by the current client is obtained by the client according to the data written into the target block;
calculating a second check value according to the data currently stored in the target block;
and if the first check value is the same as the second check value, the target block is stored.
Optionally, the method further comprises:
and the state updating module is used for updating the current state of the target block to finish storage.
The device can execute the method provided by all the embodiments of the invention, and has the corresponding functional modules and beneficial effects of executing the method. Technical details not described in detail in this embodiment can be found in the methods provided in all the foregoing embodiments of the invention.
Example III
Fig. 3 is a schematic structural diagram of a computer device according to a third embodiment of the present invention. FIG. 3 illustrates a block diagram of a computer device 312 suitable for use in implementing embodiments of the present invention. The computer device 312 shown in fig. 3 is merely an example and should not be construed as limiting the functionality and scope of use of embodiments of the present invention. Device 312 is typically a computing device that assumes file storage functionality.
As shown in FIG. 3, computer device 312 is in the form of a general purpose computing device. Components of computer device 312 may include, but are not limited to: one or more processors 316, a storage device 328, and a bus 318 that connects the different system components (including the storage device 328 and the processor 316).
The storage 328 may include computer system-readable media in the form of volatile memory, such as random access memory (Random Access Memory, RAM) 330 and/or cache memory 332. The computer device 312 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 334 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 3, commonly referred to as a "hard disk drive"). Although not shown in fig. 3, a disk drive for reading from and writing to a removable nonvolatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from and writing to a removable nonvolatile optical disk (e.g., a Compact Disc-Read Only Memory (CD-ROM), digital versatile Disc (Digital Video Disc-Read Only Memory, DVD-ROM), or other optical media), may be provided. In such cases, each drive may be coupled to bus 318 through one or more data medium interfaces. Storage 328 may include at least one program product having a set (e.g., at least one) of program modules configured to carry out the functions of embodiments of the invention.
The computer device 312 may also communicate with one or more external devices 314 (e.g., keyboard, pointing device, camera, display 324, etc.), one or more devices that enable a user to interact with the computer device 312, and/or any devices (e.g., network card, modem, etc.) that enable the computer device 312 to communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 322. Moreover, the computer device 312 may also communicate with one or more networks such as a local area network (Local Area Network, LAN), a wide area network Wide Area Network, a WAN) and/or a public network such as the internet via the network adapter 320. As shown, network adapter 320 communicates with other modules of computer device 312 via bus 318. It should be appreciated that although not shown, other hardware and/or software modules may be used in connection with computer device 312, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, disk array (Redundant Arrays of Independent Disks, RAID) systems, tape drives, data backup storage systems, and the like.
The processor 316 executes various functional applications and data processing by running programs stored in the storage 328, for example, implementing the file storage method provided by the above-described embodiment of the present invention.
Example IV
The sixth embodiment of the present invention also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements a file storage method as provided by the embodiments of the present invention.
Of course, the computer-readable storage medium provided by the embodiments of the present invention, on which the computer program stored, is not limited to the method operations described above, but may also perform the related operations in the file storage method provided by any of the embodiments of the present invention.
The computer storage media of embodiments of the invention may take the form of any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
Note that the above is only a preferred embodiment of the present invention and the technical principle applied. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, while the invention has been described in connection with the above embodiments, the invention is not limited to the embodiments, but may be embodied in many other equivalent forms without departing from the spirit or scope of the invention, which is set forth in the following claims.
Claims (10)
1. A method of storing a file, comprising:
creating a plurality of storage blocks with set sizes according to the sizes of files to be stored, and generating index files of the files to be stored according to the information of the storage blocks;
scanning storage block information in the index file according to a storage request of at least one client, and determining at least one target block meeting storage conditions;
respectively storing the data with the set size in the file to be stored into a target block;
determining that the target block is stored completely, returning to execute the operation of scanning the storage block information in the index file according to the storage request of at least one client, and determining at least one target block meeting the storage condition until all the storage blocks are determined to be stored completely;
wherein when the file to be stored is stored by at least two clients, each client corresponds to a target block respectively,
correspondingly, the storing the data with the set size in the file to be stored in the target block respectively includes:
aiming at each client, taking each client as a current client, and calling a write-in interface input parameter based on the current client;
judging whether the target block meets a storage condition or not;
if yes, writing the data with the set size in the file to be stored into the target block;
if not, determining that the target block is being written by other clients, and re-executing the step of determining the target block of the current client.
2. The method of claim 1, wherein storing information for a block comprises: the number of the memory block, the current state of the memory block, the client identification information of the most recent data writing to the memory block, the time of the most recent data writing to the memory block, the location of the memory block.
3. The method of claim 2, wherein creating a plurality of storage blocks of a set size according to a size of a file to be stored, and generating an index file of the file to be stored according to information of the plurality of storage blocks, comprises:
receiving a creation request sent by a client, wherein the creation request comprises the size and the hash value of a file to be stored;
inquiring whether an index file of the file to be stored is stored in a key value database according to the size and the hash value of the file to be stored;
if not, determining a storage area according to the size of the file to be stored;
dividing the storage area into a plurality of storage blocks with set sizes, and acquiring information of the storage blocks;
and generating an index file of the file to be stored according to the information of the storage blocks.
4. The method of claim 3, further comprising, after generating the index file of the file to be stored based on the information of the plurality of storage blocks:
and storing the size of the file to be stored, the hash value and the index file into a key value database.
5. The method of claim 2, wherein the storage conditions include: in the process of initializing or uploading the current state of the storage block, the identification information of the client terminal which recently writes data into the storage block is empty, and the time for the data to be recently written into the storage block does not exceed the set time from the current time.
6. The method of claim 2, wherein determining that the target block completes storage comprises:
judging whether the client identification information of the target block which is used for writing data into the storage block recently is the same as the identification information of the current client,
if the data are the same, a first check value sent by the current client is obtained by the client according to the data written into the target block;
calculating a second check value according to the data currently stored in the target block;
and if the first check value is the same as the second check value, the target block is stored.
7. The method of claim 1, further comprising, after determining that the target block is complete to store:
and updating the current state of the target block to finish storage.
8. A file storage device, comprising:
the storage block creating module is used for creating a plurality of storage blocks with set sizes according to the sizes of files to be stored and generating index files of the files to be stored according to the information of the storage blocks;
the target block determining module is used for determining at least one target block meeting the storage condition according to the storage block information in the storage block information scanning index file of the storage request of at least one client;
the data storage module is used for respectively storing the data with the set size in the file to be stored into the target block;
the storage completion determining module is used for determining that the target blocks are stored, returning to execute the operation of scanning the storage block information in the index file according to the storage request of at least one client, and determining at least one target block meeting the storage condition until all the storage blocks are determined to be stored;
wherein when the file to be stored is stored by at least two clients, each client corresponds to a target block respectively,
correspondingly, the data storage module is specifically configured to:
aiming at each client, taking each client as a current client, and calling a write-in interface input parameter based on the current client;
judging whether the target block meets a storage condition or not;
if yes, writing the data with the set size in the file to be stored into the target block;
if not, determining that the target block is being written by other clients, and re-executing the step of determining the target block of the current client.
9. A file storage device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any of claims 1-7 when executing the program.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811627718.8A CN111382123B (en) | 2018-12-28 | 2018-12-28 | File storage method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811627718.8A CN111382123B (en) | 2018-12-28 | 2018-12-28 | File storage method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111382123A CN111382123A (en) | 2020-07-07 |
| CN111382123B true CN111382123B (en) | 2023-06-16 |
Family
ID=71218117
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811627718.8A Active CN111382123B (en) | 2018-12-28 | 2018-12-28 | File storage method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111382123B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111949710B (en) * | 2020-08-17 | 2024-03-22 | 北京锐安科技有限公司 | Data storage method, device, server and storage medium |
| CN113094374A (en) * | 2021-04-27 | 2021-07-09 | 广州炒米信息科技有限公司 | Distributed storage and retrieval method and device and computer equipment |
| CN113704176B (en) * | 2021-07-09 | 2023-10-31 | 奇安信科技集团股份有限公司 | File scanning method, device, electronic equipment and storage medium |
| CN113468128B (en) * | 2021-07-21 | 2023-02-17 | 上海浦东发展银行股份有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN114237508A (en) * | 2021-12-16 | 2022-03-25 | 中国农业银行股份有限公司 | Data storage method, device, equipment and storage medium |
| CN114327946A (en) * | 2021-12-24 | 2022-04-12 | 北京百度网讯科技有限公司 | Shared memory access control method and device, electronic equipment and automatic driving vehicle |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107728940A (en) * | 2017-09-27 | 2018-02-23 | 柏科数据技术（深圳）股份有限公司 | The method, apparatus and storage medium of a kind of management copy in data-storage system |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9348837B2 (en) * | 2013-12-05 | 2016-05-24 | International Business Machines Corporation | Index writing in a linear tape file system |
| CN107807989B (en) * | 2017-11-03 | 2020-03-24 | 绿湾网络科技有限公司 | Small file processing method and device |
| CN109086388B (en) * | 2018-07-26 | 2020-12-29 | 百度在线网络技术（北京）有限公司 | Block chain data storage method, device, equipment and medium |
-
2018
- 2018-12-28 CN CN201811627718.8A patent/CN111382123B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107728940A (en) * | 2017-09-27 | 2018-02-23 | 柏科数据技术（深圳）股份有限公司 | The method, apparatus and storage medium of a kind of management copy in data-storage system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111382123A (en) | 2020-07-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111382123B (en) | File storage method, device, equipment and storage medium | |
| US9785664B2 (en) | Gathering transaction data associated with locally stored data files | |
| US10380103B2 (en) | Object data updating method and apparatus in an object storage system | |
| US11829624B2 (en) | Method, device, and computer readable medium for data deduplication | |
| CN109508326B (en) | Method, device and system for processing data | |
| CN111949710B (en) | Data storage method, device, server and storage medium | |
| US11314451B2 (en) | Method and apparatus for storing data | |
| CN110705235B (en) | Information input method and device for business handling, storage medium and electronic equipment | |
| CN110019873B (en) | Face data processing method, device and equipment | |
| US10917484B2 (en) | Identifying and managing redundant digital content transfers | |
| CN111338834B (en) | Data storage method and device | |
| US11204926B2 (en) | Storing partial tuples from a streaming application in a database system | |
| CN107862035B (en) | Network reading method and device for conference record, intelligent tablet and storage medium | |
| CN113190384A (en) | Data recovery control method, device, equipment and medium based on erasure codes | |
| CN107992457B (en) | Information conversion method, device, terminal equipment and storage medium | |
| CN111008934B (en) | Scene construction method, device, equipment and storage medium | |
| US11269848B2 (en) | Preventing unnecessary upload | |
| CN113810477A (en) | File uploading method, device, equipment and storage medium | |
| CN112925796A (en) | Write consistency control method, device, equipment and storage medium | |
| CN114285838A (en) | File uploading method, device, system, storage medium and electronic equipment | |
| CN112804312B (en) | File uploading method, equipment and computer readable medium | |
| CN112148705A (en) | Data migration method and device | |
| US11379147B2 (en) | Method, device, and computer program product for managing storage system | |
| US11842077B2 (en) | Method, device, and computer program product for transmitting data for object storage | |
| US20210081110A1 (en) | Method, device and computer program product for managing storage blocks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20231008 Address after: 31a, 15 / F, building 30, maple mall, bangrang Road, Brazil, Singapore Patentee after: Baiguoyuan Technology (Singapore) Co.,Ltd. Address before: 511400 floor 23-39, building B-1, Wanda Plaza North, Wanbo business district, 79 Wanbo 2nd Road, Nancun Town, Panyu District, Guangzhou City, Guangdong Province Patentee before: GUANGZHOU BAIGUOYUAN INFORMATION TECHNOLOGY Co.,Ltd. |