CN113971163A - Small file merging storage method, small file reading method and server - Google Patents
Small file merging storage method, small file reading method and server Download PDFInfo
- Publication number
- CN113971163A CN113971163A CN202010718204.4A CN202010718204A CN113971163A CN 113971163 A CN113971163 A CN 113971163A CN 202010718204 A CN202010718204 A CN 202010718204A CN 113971163 A CN113971163 A CN 113971163A
- Authority
- CN
- China
- Prior art keywords
- small file
- large object
- small
- metadata
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 69
- 238000004806 packaging method and process Methods 0.000 claims description 4
- 230000009467 reduction Effects 0.000 abstract description 5
- 238000010586 diagram Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 5
- 230000009286 beneficial effect Effects 0.000 description 4
- 238000004590 computer program Methods 0.000 description 4
- 238000007726 management method Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 230000001186 cumulative effect Effects 0.000 description 3
- 238000013500 data storage Methods 0.000 description 3
- 238000010295 mobile communication Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 101100226364 Arabidopsis thaliana EXT1 gene Proteins 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 238000007596 consolidation process Methods 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 238000004880 explosion Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
- G06F16/137—Hash-based
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of storage, and discloses a small file merging storage method, a small file reading method and a server. The method comprises the following steps: acquiring a write request of a small file; confirming a large object corresponding to the write request; writing the data of the small file into the large object; the metadata of the small files and the metadata of the large objects corresponding to the small files are updated to the index pool, so that the problem of reduction of storage performance of the large amount of small files can be solved, and stable writing performance is kept.
Description
Technical Field
The invention relates to the technical field of storage, in particular to a small file merging storage method, a small file reading method and a server.
Background
In the information explosion era, data generated every day has the remarkable characteristics of huge storage capacity, various data types, large data size change, fast flow and the like, and can generate tens of millions, hundreds of millions, billions and billions of massive small files.
Traditional file systems, including local file systems, distributed file systems, and object storage systems, are designed for large files, such as XFS/EXT4, GlusterFS, GFS, HDFS, and have significant files on the implementation strategies such as metadata management, data layout, cache management, etc., while the application of large numbers of small files has been greatly reduced in terms of performance and storage efficiency.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a small file merging and storing method, a small file reading method, and a server, which can solve the problem of the storage performance degradation of a large number of small files.
In a first aspect, an embodiment of the present invention provides a method for merging and storing small files, where the method includes:
acquiring a write request of a small file;
confirming a large object corresponding to the write request;
writing the data of the small file into the large object;
and updating the metadata of the small file and the metadata of the large object corresponding to the small file to an index pool.
In some embodiments, the write request carries a name of the small file;
the confirming the large object corresponding to the write request comprises:
and carrying out hash operation on the name of the small file to obtain a hash value, and determining a target large object corresponding to the small file according to the hash value.
In some embodiments, the determining a target large object corresponding to the small file according to the hash value includes:
and if a large object corresponding to the hash value is found and the number or the accumulated size of the small files in the large object does not exceed a preset threshold value, taking the large object as a target large object of the small files.
In some embodiments, the method further comprises:
if a large object corresponding to the hash value is found, and the number or the accumulated size of the small files in the large object exceeds a preset threshold value; or,
if the large object corresponding to the hash value is not found;
a new large object is generated as the target large object for the small file.
In some embodiments, the writing the data of the small file to the large object includes:
acquiring the offset of a large object corresponding to the small file;
writing data of the small file to an offset of the large object.
In some embodiments, the write request also carries a space pre-allocation flag,
the method further comprises the following steps:
and allocating a fixed space in the large object according to the space pre-allocation mark.
In a second aspect, an embodiment of the present invention further provides a method for reading a small file, where the method includes:
acquiring a reading request of a small file, wherein the reading request carries the name of the small file;
acquiring metadata of the small file according to the name of the small file;
acquiring data of the small file from the large object according to the metadata of the small file;
and packaging the data of the small files and the metadata of the small files and returning the packaged data of the small files and the metadata of the small files to the terminal equipment.
In some embodiments, the metadata of the small file carries index information of a large object corresponding to the small file;
the acquiring data of the small file from the large object according to the metadata of the small file comprises the following steps:
determining the name and offset of the corresponding large object according to the index information;
and acquiring data of the small file from the large object according to the name and the offset of the large object.
In a third aspect, an embodiment of the present invention further provides a server, including:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the above method steps.
In a fourth aspect, the present invention also provides a non-transitory computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, cause the processor to perform the above method steps.
Compared with the prior art, the invention has the beneficial effects that: different from the situation of the prior art, the small file merging storage method, the small file reading method and the server in the embodiments of the present invention obtain the write request of the small file, then confirm the large object corresponding to the write request, write the data of the small file into the large object, and update the metadata of the small file and the metadata of the large object corresponding to the small file into the index pool, thereby solving the problem of the reduction of the storage performance of the massive small files and maintaining the stable write performance.
Drawings
One or more embodiments are illustrated by way of example in the accompanying drawings, which correspond to the figures in which like reference numerals refer to similar elements and which are not to scale unless otherwise specified.
FIG. 1 is a schematic diagram of an application scenario of a small file merging storage method and a small file reading method according to an embodiment of the present invention;
FIG. 2 is a flow chart illustrating a method for merging and storing small files according to an embodiment of the present invention;
FIG. 3 is a flow chart illustrating a method for reading a small file according to an embodiment of the present invention;
FIG. 4 is a detailed diagram of a method for merging and storing small files according to an embodiment of the present invention;
FIG. 5 is a detailed diagram of a method for reading a small file according to an embodiment of the present invention;
FIG. 6 is a schematic structural diagram of a small file merge storage apparatus according to an embodiment of the present invention;
FIG. 7 is a schematic structural diagram of a small document reading apparatus according to an embodiment of the present invention;
fig. 8 is a schematic diagram of a hardware configuration of a server in an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that, if not conflicted, the various features of the embodiments of the invention may be combined with each other within the scope of protection of the invention. Additionally, while functional block divisions are performed in apparatus schematics, with logical sequences shown in flowcharts, in some cases, steps shown or described may be performed in sequences other than block divisions in apparatus or flowcharts. The terms "first", "second", "third", and the like used in the present invention do not limit data and execution order, but distinguish the same items or similar items having substantially the same function and action.
The small file merging storage method and the small file reading method provided by the embodiment of the invention can be applied to the application environment shown in fig. 1, wherein the application environment comprises terminal equipment and a server. Fig. 1 schematically shows a terminal device 10 communicating with a server 20 via a network. The terminal device 10 may be, but is not limited to, a personal computer, a notebook computer, a smart phone, a tablet computer, and the like. The server 20 may be, but not limited to, a rack server, a blade server, a tower server, or a rack server, and may also be a server cluster composed of a plurality of servers, or a cloud computing service center.
The server is provided with an object storage gateway layer, a storage service layer, a metadata center cluster, a data storage cluster, a data pool and an index pool. The object storage gateway layer is used for receiving a terminal device request, segmenting small files according to the size of a specified block, and sending each block to a specified storage node according to a routing rule of a meta-service; the metadata center cluster is used for maintaining the state of the whole cluster and a routing rule, and the routing rule is the corresponding relation between an object and a storage node; the data storage cluster is used for storing the small files and is responsible for consistency among the copies; the data pool is used for storing actual data of the small files; the index pool is used for storing the metadata of the small files and the metadata of the large objects.
It should be noted that the method provided by the embodiment of the present invention may be further extended to other suitable application environments, and is not limited to the application environment described in fig. 1. In an actual application process, the application environment may include more or fewer terminal devices and servers.
As shown in fig. 2, an embodiment of the present invention provides a method for merging and storing small files, where the method includes:
In the embodiment of the present invention, the write request is a request that the terminal device needs to store the small file. Specifically, an object storage gateway layer on the server acquires a write request of a small file sent by the terminal device.
And step 204, confirming the large object corresponding to the write request.
In an embodiment of the present invention, the large object is used for storing the small file. After acquiring a write-in request of a small file sent by terminal equipment, an object storage gateway layer confirms a large object corresponding to the write-in request according to the write-in request.
In other embodiments, as an implementation manner of step 204, the name of the small file is carried in the write request sent by the terminal device, the object storage gateway layer performs hash operation according to the name of the small file to obtain a hash value, and determines a target large object corresponding to the small file according to the hash value, that is, the obtained hash value may specifically correspond to a large object. It should be noted that the memory of the server stores the index relationship between the hash value and the large object in advance.
In some other embodiments, determining the target large object corresponding to the small file from the hash value comprises: and if the object corresponding to the hash value is found and the number or the accumulated size of the small files written by the large object does not exceed a preset threshold value, taking the large object as a target large object of the small files.
It is understood that, in some other embodiments, if a large object corresponding to the hash value is found and the number or the cumulative size of the small files written by the large object exceeds a preset threshold, a large object is newly generated as the target large object of the small files. Or, a large object corresponding to the hash value is not found in the memory, and a large object is regenerated as a target large object of the small file. Note that when a large object is newly generated, the offset is written from 0. Wherein an offset may be understood as a storage location of a small file in a large object. Illustratively, if the large object is a newly generated target large object, then the small file is written starting from the position where the target large object offset is 0.
In other embodiments, a fixed storage space may also be allocated for small files in large objects. Specifically, when the small file write request further carries a space pre-allocation flag, the storage service layer pre-allocates the space of the large object according to the space pre-allocation flag, that is, allocates a fixed space in the large object according to the space pre-allocation flag for storing the small file data. The space pre-allocation flag may be 0 or 1, which is only an example and is not a limitation.
And after determining the target large object corresponding to the small file according to the hash value, directly writing the data of the small file into the large object.
In other embodiments, as an implementation manner of step 206, since the large object corresponding to the small file is already determined in advance, the offset of the large object corresponding to the small file may be obtained, and after the offset of the large object is obtained, the data of the small file is directly written into the offset of the large object.
And step 208, updating the metadata of the small file and the metadata of the large object corresponding to the small file to an index pool.
In the embodiment of the invention, after the data of the small file is written into the large object, the metadata of the small file and the metadata of the large object are obtained, and the metadata of the small file and the metadata of the large object are updated to an index pool. The metadata of the small file carries the name of the object, and the metadata of the large object carries the storage information of the small file, that is, how much data of the small file is stored on the current large object and how much write offset of the small file in the current large object is recorded in the metadata of the large object.
In some other embodiments, after the metadata of the small file and the metadata of the large object corresponding to the small file are updated to the index pool, information of successful storage is returned to the terminal device, so that the terminal device can know whether the small file is successfully stored.
In the embodiment of the invention, the writing request of the small file is obtained, the large object corresponding to the writing request is confirmed, the data of the small file is written into the large object, and the metadata of the small file and the metadata of the large object corresponding to the small file are updated to the index pool, so that the problem of the reduction of the storage performance of the mass small files can be solved, and the writing performance is kept stable.
As shown in fig. 3, an embodiment of the present invention further provides a method for reading a small file, where the method includes:
In the embodiment of the invention, the reading request is a request sent by the terminal equipment for reading the small file. Specifically, the object storage gateway layer obtains a read request for obtaining the small file sent by the terminal device, where the read request carries a name of the small file.
And 304, acquiring the metadata of the small file according to the name of the small file.
Specifically, since the metadata of the small file is stored in the index pool in advance, after the object storage gateway layer obtains the read request sent by the terminal device, the object storage gateway layer obtains the metadata of the small file from the index pool according to the name of the small file.
The large object stores the data of the small file, so the object storage gateway layer acquires the data of the small file from the large object according to the metadata of the small file.
In other embodiments, as an implementation manner of step 306, the metadata of the small file carries index information of a large object corresponding to the small file, so that after the object storage gateway layer obtains the index information, the object storage gateway layer obtains a name and an offset of the corresponding large object from an index pool according to the index information, and then obtains data of the small file from the large object in a data pool according to the name and the offset of the large object.
And 308, packaging the data of the small file and the metadata of the small file and returning the packaged data of the small file to the terminal equipment.
Specifically, after the object storage gateway layer obtains the data of the small file, the data of the small file and the metadata of the small file are packaged, namely assembled and returned to the terminal device.
In the embodiment of the invention, the reading performance under the storage of massive small files can be improved by acquiring the reading request of the small files, wherein the reading request carries the names of the small files, acquiring the metadata of the small files according to the names of the small files, then acquiring the data of the small files from the large object according to the metadata of the small files, and finally packaging the data of the small files and the metadata of the small files and returning the packaged data of the small files and the metadata of the small files to the terminal equipment.
To facilitate understanding of the present invention, the following will specifically describe the merging and storing of small files:
as shown in fig. 4, first, the terminal device sends a write request of a small file to the server, where the write request carries a name of the small file, and an object storage gateway layer on the server obtains the write request of the small file sent by the terminal device and obtains the name of the small file carried in the write request.
And then, the object storage gateway layer performs hash operation according to the name of the small file to obtain a hash value, and determines a target large object corresponding to the small file according to the hash value. And determining the target large object corresponding to the small file according to the hash value, wherein the target large object is divided into three conditions. In the first case, if an object corresponding to the hash value is found and the number or the cumulative size of the small files in the large object does not exceed a preset threshold, the large object is used as a target large object of the small files. In the second case, if a large object corresponding to the hash value is found and the number or the cumulative size of the small files in the large object exceeds a preset threshold, a new large object is generated as the target large object of the small files. In a third case, if the large object corresponding to the hash value is not found in the memory, a new large object is generated as the target large object of the small file. If the large object is a target large object to be newly generated, the small file is written from a position offset from the target large object by 0.
Further, when the small file write request also carries a space pre-allocation flag, the storage service layer pre-allocates the space of the large object according to the space pre-allocation flag, that is, a fixed space is allocated in the large object according to the space pre-allocation flag for storing the small file data.
And then, after determining a target large object corresponding to the small file according to the hash value, acquiring the offset of the large object corresponding to the small file, and directly writing the data of the small file into the offset of the object after acquiring the offset of the large object.
Further, after the data of the small file is written into the large object, the metadata of the small file and the metadata of the large object are obtained, and the metadata of the small file and the metadata of the large object are updated to the index pool together, so that the problem of reduction of storage performance of the mass small files can be solved, and the writing performance is kept stable.
And finally, returning the information of successful writing to the terminal equipment so that the terminal equipment can know whether the small file is successfully stored or not.
To facilitate understanding of the present invention, the following description will be made specifically for small file reading:
as shown in fig. 5, first, a terminal device sends a request for reading a small file to a server, where the request for reading the small file carries a name of the small file, and an object storage gateway layer on the server obtains the request for reading the small file sent by the terminal device and obtains the name of the small file in the request for reading.
And then, the object storage gateway layer acquires the metadata of the small files from the index pool according to the names of the small files.
Further, the object storage gateway layer acquires the data of the small file from the large object according to the metadata of the small file. Specifically, since the metadata of the small file carries the index information of the large object corresponding to the small file, after the object storage gateway layer obtains the index information, the object storage gateway layer obtains the name and offset of the corresponding large object from the index pool according to the index information, and then obtains the data of the small file from the large object in the data pool according to the name and offset of the large object.
Further, after the object storage gateway layer obtains the data of the small file, the data of the small file and the metadata of the small file are packaged, namely assembled and returned to the terminal device, so that the reading performance of the small file can be improved.
Correspondingly, an embodiment of the present invention further provides a small file merging storage apparatus 600, as shown in fig. 6, including:
an obtaining module 602, configured to obtain a write request of a small file;
a confirming module 604, configured to confirm the large object corresponding to the write request;
a writing module 606, configured to write data of the small file into the large object;
an updating module 608, configured to update the metadata of the small file and the metadata of the large object corresponding to the small file to an index pool.
In the embodiment of the invention, the writing request of the small file is acquired by the acquisition module, the large object corresponding to the writing request is determined by the confirmation module, the data of the small file is written into the large object by the writing module, and the metadata of the small file and the metadata of the large object corresponding to the small file are updated to the index pool by the updating module, so that the problem of the reduction of the storage performance of the mass small files can be solved, and the writing performance of the small files can be maintained.
It should be noted that the above small file merging storage device can execute the small file merging storage method provided in the embodiments of the present invention, and has corresponding functional modules and beneficial effects of the execution method.
Correspondingly, an embodiment of the present invention further provides a small file reading apparatus 700, as shown in fig. 7, including:
a first obtaining module 702, configured to obtain a read request for a small file, where the read request carries a name of the small file;
a second obtaining module 704, configured to obtain metadata of the small file according to the name of the small file;
a third obtaining module 706, configured to obtain data of a small file from the large object according to the metadata of the small file;
a sending module 708, configured to package and return the data of the small file and the metadata of the small file to the terminal device.
In the embodiment of the invention, a reading request of a small file is obtained through a first obtaining module, the reading request carries the name of the small file, then, metadata of the small file is obtained through a second obtaining module according to the name of the small file, then, the data of the small file is obtained from the large object through a third obtaining module according to the metadata of the small file, and finally, the data of the small file and the metadata of the small file are packaged and returned to the terminal equipment through a returning module, so that the reading performance under the storage of massive small files can be improved.
It should be noted that the above small file reading apparatus can execute the small file reading method provided in the embodiments of the present invention, and has corresponding functional modules and beneficial effects of the execution method.
Fig. 8 is a schematic diagram of a hardware structure of a server according to an embodiment of the present invention. As shown in fig. 8, the server 800 includes:
one or more processors 802 and a memory 804, one processor 802 being illustrated in fig. 8.
The processor 802 and the memory 804 may be connected by a bus or other means, such as by a bus in FIG. 8.
The memory 804 serves as a non-volatile computer-readable storage medium and may be used for storing non-volatile software programs, non-volatile computer-executable programs, and modules, such as program instructions/modules corresponding to the small file merging storage method and the small file reading method in the embodiment of the present invention (for example, the obtaining module 602, the confirming module 604, the writing module 606, and the updating module 608 shown in fig. 6; and the first obtaining module 702, the second obtaining module 704, the third obtaining module 706, and the sending module 708 shown in fig. 7). The processor 802 executes various functional applications and data processing of the server by running the nonvolatile software programs, instructions, and modules stored in the memory 804, that is, implements the small file merge storage method and the small file reading method in the above embodiments.
The memory 804 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created from use by the small-file consolidation storage means and the small-file reading means, and the like. Further, the memory 804 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 804 optionally includes memory located remotely from the processor 802, which may be connected to the doclet merge storage and the doclet read device via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The one or more modules are stored in the memory 804, and when executed by the one or more servers, perform the doclet merge storage method and the doclet reading method in any of the above-described method embodiments, for example, performing the above-described method steps 202 to 208 in fig. 2, and the method steps 302 to 308 in fig. 3; the functions of blocks 602 to 608 in fig. 6, and the functions of blocks 702 to 708 in fig. 7 are implemented.
Embodiments of the present invention also provide a computer program product, including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions that, when executed by a computer, cause the computer to perform: method steps 202 to 208 in fig. 2, and method steps 302 to 308 in fig. 3.
The product can execute the method provided by the embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method. For technical details that are not described in detail in this embodiment, reference may be made to the method provided by the embodiment of the present invention.
The terminal device of the embodiment of the invention exists in various forms, including but not limited to:
(1) mobile communication devices, which are characterized by mobile communication capabilities and are primarily targeted at providing voice and data communications. Such terminals include smart phones (e.g., iphones), multimedia phones, functional phones, and low-end phones, among others.
(2) The ultra-mobile personal computer equipment belongs to the category of personal computers, has calculation and processing functions and generally has the characteristic of mobile internet access. Such terminals include PDA, MID, and UMPC devices, such as ipads.
(3) Portable entertainment devices such devices may display and play multimedia content. Such devices include audio and video players (e.g., ipods), handheld game consoles, electronic books, as well as smart toys and portable car navigation devices.
The server of embodiments of the present invention exists in a variety of forms, including but not limited to:
(1) tower server
The general tower server chassis is almost as large as the commonly used PC chassis, while the large tower chassis is much larger, and the overall dimension is not a fixed standard.
(2) Rack-mounted server
Rack-mounted servers are a type of server that has a standard width of 19 inch racks, with a height of from 1U to several U, due to the dense deployment of the enterprise. Placing servers on racks not only facilitates routine maintenance and management, but also may avoid unexpected failures. First, placing the server does not take up too much space. The rack servers are arranged in the rack in order, and no space is wasted. Secondly, the connecting wires and the like can be neatly stored in the rack. The power line, the LAN line and the like can be distributed in the cabinet, so that the connection lines accumulated on the ground can be reduced, and the accidents such as the electric wire kicking off by feet can be prevented. The specified dimensions are the width (48.26cm ═ 19 inches) and height (multiples of 4.445 cm) of the server. Because of its 19 inch width, a rack that meets this specification is sometimes referred to as a "19 inch rack".
(3) Blade server
A blade server is a HAHD (High Availability High Density) low cost server platform designed specifically for the application specific industry and High Density computer environment, where each "blade" is actually a system motherboard, similar to an individual server. In this mode, each motherboard runs its own system, serving a designated group of different users, without any relationship to each other. Although system software may be used to group these motherboards into a server cluster. In the cluster mode, all motherboards can be connected to provide a high-speed network environment, and resources can be shared to serve the same user group.
(4) Cloud server
The cloud server (ECS) is a computing Service with simplicity, high efficiency, safety, reliability, and flexible processing capability. The management mode is simpler and more efficient than that of a physical server, and a user can quickly create or release any plurality of cloud servers without purchasing hardware in advance. The distributed storage of the cloud server is used for integrating a large number of servers into a super computer, and a large number of data storage and processing services are provided. The distributed file system and the distributed database allow access to common storage resources, and IO sharing of application data files is achieved. The virtual machine can break through the limitation of a single physical machine, dynamically adjust and allocate resources to eliminate single-point faults of the server and the storage equipment, and realize high availability.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a general hardware platform, and certainly can also be implemented by hardware. It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware related to instructions of a computer program, which can be stored in a computer readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; within the idea of the invention, also technical features in the above embodiments or in different embodiments may be combined, steps may be implemented in any order, and there are many other variations of the different aspects of the invention as described above, which are not provided in detail for the sake of brevity; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. A method for merging and storing small files is characterized by comprising the following steps:
acquiring a write request of a small file;
confirming a large object corresponding to the write request;
writing the data of the small file into the large object;
and updating the metadata of the small file and the metadata of the large object corresponding to the small file to an index pool.
2. The method of claim 1, wherein the write request carries a name of the small file;
the confirming the large object corresponding to the write request comprises:
and carrying out hash operation on the name of the small file to obtain a hash value, and determining a target large object corresponding to the small file according to the hash value.
3. The method of claim 2, wherein the determining the target large object corresponding to the small file according to the hash value comprises:
and if a large object corresponding to the hash value is found and the number or the accumulated size of the small files in the large object does not exceed a preset threshold value, taking the large object as a target large object of the small files.
4. The method of claim 3, further comprising:
if a large object corresponding to the hash value is found, and the number or the accumulated size of the small files in the large object exceeds a preset threshold value; or,
if the large object corresponding to the hash value is not found;
a new large object is generated as the target large object for the small file.
5. The method according to any of claims 1-4, wherein said writing data of said small file to said large object comprises:
acquiring the offset of a large object corresponding to the small file;
writing data of the small file to an offset of the large object.
6. The method of claim 5, wherein the write request further carries a space pre-allocation flag,
the method further comprises the following steps:
and allocating a fixed space in the large object according to the space pre-allocation mark.
7. A method for reading a small file, the method comprising:
acquiring a reading request of a small file, wherein the reading request carries the name of the small file;
acquiring metadata of the small file according to the name of the small file;
acquiring data of the small file from the large object according to the metadata of the small file;
and packaging the data of the small files and the metadata of the small files and returning the packaged data of the small files and the metadata of the small files to the terminal equipment.
8. The method according to claim 7, wherein the metadata of the small file carries index information of a large object corresponding to the small file;
the acquiring data of the small file from the large object according to the metadata of the small file comprises the following steps:
determining the name and offset of the corresponding large object according to the index information;
and acquiring data of the small file from the large object according to the name and the offset of the large object.
9. A server, comprising:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.
10. A non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010718204.4A CN113971163A (en) | 2020-07-23 | 2020-07-23 | Small file merging storage method, small file reading method and server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010718204.4A CN113971163A (en) | 2020-07-23 | 2020-07-23 | Small file merging storage method, small file reading method and server |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113971163A true CN113971163A (en) | 2022-01-25 |
Family
ID=79585371
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010718204.4A Pending CN113971163A (en) | 2020-07-23 | 2020-07-23 | Small file merging storage method, small file reading method and server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113971163A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114564149A (en) * | 2022-02-25 | 2022-05-31 | 上海英方软件股份有限公司 | Data storage method, device, equipment and storage medium |
| CN117648297A (en) * | 2024-01-30 | 2024-03-05 | 中国人民解放军国防科技大学 | Method, system, equipment and medium for offline merging of small files based on object storage |
-
2020
- 2020-07-23 CN CN202010718204.4A patent/CN113971163A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114564149A (en) * | 2022-02-25 | 2022-05-31 | 上海英方软件股份有限公司 | Data storage method, device, equipment and storage medium |
| CN114564149B (en) * | 2022-02-25 | 2024-03-26 | 上海英方软件股份有限公司 | Data storage method, device, equipment and storage medium |
| CN117648297A (en) * | 2024-01-30 | 2024-03-05 | 中国人民解放军国防科技大学 | Method, system, equipment and medium for offline merging of small files based on object storage |
| CN117648297B (en) * | 2024-01-30 | 2024-06-11 | 中国人民解放军国防科技大学 | Method, system, equipment and medium for offline merging of small files based on object storage |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3333718B1 (en) | Task allocation method and task allocation apparatus for distributed data calculation | |
| CN114064563A (en) | Data migration method and server based on object storage | |
| CN108287669A (en) | Date storage method, device and storage medium | |
| CN102938784A (en) | Method and system used for data storage and used in distributed storage system | |
| CN108614976A (en) | Authority configuring method, device and storage medium | |
| CN108255989B (en) | Picture storage method and device, terminal equipment and computer storage medium | |
| CN106817388B (en) | Method and device for acquiring data by virtual machine and host machine and system for accessing data | |
| CN113971163A (en) | Small file merging storage method, small file reading method and server | |
| CN110415042A (en) | A kind of discount coupon generates system, method and coupon server | |
| CN111064786B (en) | Account identifier management method and device | |
| CN110515728B (en) | Server scheduling method and device, electronic equipment and machine-readable storage medium | |
| CN112084173A (en) | Data migration method and device and storage medium | |
| CN112171656B (en) | Control method and device of mechanical arm and server | |
| CN110489468A (en) | A kind of deriving method of data sheet, device and server | |
| CN111803917A (en) | Resource processing method and device | |
| CN112269665B (en) | Memory processing method and device, electronic equipment and storage medium | |
| CN107644020B (en) | Data storage and calling method and device | |
| CN112597406A (en) | File transmission method and device, terminal equipment and storage medium | |
| CN111258959A (en) | Data acquisition method, data providing method and device | |
| CN105144073A (en) | Removable storage device identity and configuration information | |
| CN111404828A (en) | Method and device for realizing global flow control | |
| CN107491551B (en) | Method and device for realizing client access of distributed file system | |
| CN111104301B (en) | Method and system for judging barrier user in webpage | |
| CN114726657A (en) | Method and device for interrupt management and data receiving and sending management and intelligent network card | |
| CN110703997A (en) | Cloud storage system, data storage method, device, equipment and computer medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |