CN117667865A - Data distributed storage method, device, equipment and medium - Google Patents
Data distributed storage method, device, equipment and medium Download PDFInfo
- Publication number
- CN117667865A CN117667865A CN202311660187.3A CN202311660187A CN117667865A CN 117667865 A CN117667865 A CN 117667865A CN 202311660187 A CN202311660187 A CN 202311660187A CN 117667865 A CN117667865 A CN 117667865A
- Authority
- CN
- China
- Prior art keywords
- unused
- segment
- target
- data
- segmented
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 238000011161 development Methods 0.000 claims abstract description 38
- 238000013500 data storage Methods 0.000 claims abstract description 33
- 238000005192 partition Methods 0.000 claims description 38
- 238000004590 computer program Methods 0.000 claims description 22
- 239000013589 supplement Substances 0.000 claims description 4
- 238000012795 verification Methods 0.000 claims description 4
- 230000018109 developmental process Effects 0.000 claims 4
- 230000011218 segmentation Effects 0.000 claims 1
- 230000004044 response Effects 0.000 abstract description 4
- 238000004891 communication Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 7
- 230000008569 process Effects 0.000 description 7
- 238000012545 processing Methods 0.000 description 5
- 238000013461 design Methods 0.000 description 4
- 230000000153 supplemental effect Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 230000001627 detrimental effect Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 229910021389 graphene Inorganic materials 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a data distributed storage method, device, equipment and medium. The method comprises the following steps: a space application request is sent to a metadata server, the space application request is used for indicating the metadata server to generate a plurality of unused segmented objects at one time according to the space application request, and the unused segmented objects are issued to a third party development tool; selecting a target segment object from the local unused segment objects, and writing the data to be stored into the target segment object; if the target segment object is fully written, creating a target actual space on a corresponding segment of the data storage server according to the segment ID recorded in the target segment object; and storing the data in the target segment object into a target actual space. By adopting the method and the device, delay on the IO path can be reduced, and the writing performance and response speed of the distributed storage system can be improved.
Description
Technical Field
The present disclosure relates to the field of distributed storage technologies, and in particular, to a method, an apparatus, a device, and a medium for data distributed storage.
Background
With the popularity and depth of big data applications, the underlying computing framework presents greater challenges to storage systems in terms of scale and performance requirements. The high-performance computer has higher and higher performance requirements on the distributed file system, and the file system read-write efficiency becomes a key factor for limiting the performance of the file system in an application scene of frequent creation and deletion of massive small files and large-scale concurrent I/O operation.
A common distributed storage system consists of a data storage server (chunks server), a metadata server (master), and a third party development tool (sdk). The data storage server is used for managing real storage equipment and processing data reading and writing; the metadata server is a coordinator and a metadata storage area of distributed storage and is used for coordinating the space allocation of the storage cluster and maintaining the health of the cluster; the third party development tool is a tool for externally exposed operation distributed storage, and is used for externally calling related interfaces to perform operations such as writing, reading and the like on the storage.
At present, when a third party development tool writes data and requires application space, an interface of a metadata server is required to be called for application, the metadata server distributes specific data blocks (chunk) on a data storage server according to the decision of metadata, then the metadata server calls corresponding data blocks to distribute space, and the applied space is returned to the third party development tool for writing. Each time a write process downloads, 3 end-to-end connections and one drop operation are required, and this operation cannot be avoided every time a space write is applied, and since it is the io critical path, jitter often occurs when writing data, which is detrimental to the stability of the system.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a data distributed storage method, apparatus, device, and medium, which can reduce delay on an IO path and improve writing performance and response speed of a distributed storage system.
In a first aspect, the present application provides a data distributed storage method, including:
a space application request is sent to a metadata server, the space application request is used for indicating the metadata server to generate a plurality of unused segmented objects at one time according to the space application request, and the unused segmented objects are issued to a third party development tool;
selecting a target segment object from the local unused segment objects, and writing the data to be stored into the target segment object;
if the target segment object is fully written, creating a target actual space on a corresponding segment of the data storage server according to the segment ID recorded in the target segment object;
and storing the data in the target segment object into a target actual space.
In a second aspect, the present application provides a data distributed storage device, comprising:
the application module is used for sending a space application request to the metadata server, indicating the metadata server to generate a plurality of unused segmented objects at one time according to the space application request, and issuing the plurality of unused segmented objects to a third party development tool;
the writing module is used for selecting a target segmented object from the local unused segmented objects and writing the data to be stored into the target segmented object;
the creating module is used for creating a target actual space on a corresponding partition of the data storage server according to the partition ID recorded in the target segment object if the target segment object is fully written;
and the storing module is used for storing the data in the target segment object into the target actual space.
In a third aspect, the present application provides a computer device comprising a memory storing a computer program and a processor implementing the steps of the method described above when the processor executes the computer program.
In a fourth aspect, the present application provides a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of the method described above.
In a fifth aspect, the present application provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method described above.
The data distributed storage method, apparatus, computer device, computer readable storage medium, and computer program product described above requests more unused segment objects by requiring communication with a metadata server when the number of unused segment objects is insufficient for a third party development tool. Compared with the interaction with the metadata server in each writing, the design reduces network overhead and saves bandwidth and resource consumption. By creating multiple unused segmented objects in readiness at once and requesting more unused segmented objects from the metadata server when needed, the system can dynamically expand the available space as needed. The design can adapt to the increasing data volume and the requirements of users, and the expandability of the system is maintained.
Drawings
Fig. 1 is an application environment diagram of a data distributed storage method provided in an embodiment of the present application;
fig. 2 is a flow chart of a data distributed storage method according to an embodiment of the present application;
FIG. 3 is a block diagram of a data distributed storage device according to an embodiment of the present disclosure;
FIG. 4 is an internal block diagram of a computer device according to an embodiment of the present application;
FIG. 5 is an internal block diagram of another computer device according to an embodiment of the present application;
fig. 6 is an internal structural diagram of a computer-readable storage medium according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.
The data distributed storage method provided by the embodiment of the application can be applied to an application environment shown in fig. 1. Wherein, the third party development tool 101 stores data on corresponding blocks in the data storage server 103 through unused segmented objects issued by the metadata server 102.
As shown in fig. 2, an embodiment of the present application provides a data distributed storage method, and the method is applied to the third party development tool 101 in fig. 1 for explanation. It is understood that the computer device may include at least one of a terminal and a server. The method comprises the following steps:
and S201, a space application request is sent to the metadata server, the space application request is used for indicating the metadata server to generate a plurality of unused segmented objects at one time according to the space application request, and the unused segmented objects are issued to a third party development tool.
S202, selecting a target segment object from the local unused segment objects, and writing the data to be stored into the target segment object.
And S203, if the target segment object is fully written, creating a target actual space on a corresponding segment of the data storage server according to the segment ID recorded in the target segment object.
S204, storing the data in the target segment object into a target actual space.
Wherein the unused state segmented objects are recorded only in the database of the third party development tool and no real space is created on the corresponding block of the data storage server.
Specifically, the third party development tool sends a space application request to the metadata server, and after receiving the space application request, the metadata server creates a plurality of Segment (Segment) objects at one time, sets the states of the Segment objects to be unused, and sends the unused Segment objects to the third party development tool.
The third party development tool places the unused segmented objects in a local database, selects one unused segmented object from the local database as a target segmented object when the data to be stored needs to be written into the distributed storage system, and sets the state of the unused segmented object as in-use. And storing the reference of the target segment object in a local cache, and temporarily writing the data to be stored.
After the target segment object is fully written, the third party development tool creates a target actual space on the corresponding segment of the data storage server according to the segment ID recorded in the target segment object. And the third party development tool stores the data to be stored, which is temporarily written in the target segment object, into a target actual space to finish the actual storage of the data to be stored in the distributed storage system.
Therefore, in the embodiment of the application, by creating a plurality of unused segment objects at one time, the third party development tool can directly acquire available segment objects from the local cache for writing without frequently communicating with the metadata server, so that delay on an IO path can be reduced, and writing performance and response speed can be improved; meanwhile, the management process of the segmented object is removed from the metadata server, so that the load and pressure of the metadata server are reduced, the metadata server can process metadata management and other core tasks more intensively, and the overall system performance and reliability are improved; in addition, since the third party development tool can acquire the unused segmented object from the local cache for writing, instant allocation of the metadata server is not needed, concurrent writing operation can be supported, and a plurality of third party development tool instances can simultaneously utilize the unused segmented object for writing, so that overall concurrency performance is improved.
In some embodiments, selecting the target segment object from the local unused segment objects includes:
adding a plurality of unused segmented objects issued by the metadata server to a local unused segmented object list;
selecting an unused segment object from the unused segment object list as a target segment object;
correspondingly, the method further comprises the steps of:
and when the number of unused segmented objects in the unused segmented object list is lower than the supplement threshold value, re-sending a space application request to the metadata server.
In particular, a list is used to store unused segmented objects for quick allocation of space. For unused segmented objects delivered by the metadata server, the third party development tool adds them to the list of unused segmented objects in the local database. When the number of unused segment objects in the unused segment object list is sufficient, the third party development tool normally extracts an unused segment object from the unused segment object list as a target segment object; when the number of unused segment objects in the unused segment object list is insufficient (below the replenishment threshold), the third party development tool resends a space application request to the metadata server to obtain a new unused segment object from the metadata server to replenish the unused segment object list.
It should be noted that the supplemental threshold is not a fixed value and may be dynamically adjusted as the load conditions of portions or the whole of the distributed storage system. For example, when the metadata server is operating normally, the supplemental threshold may be set to half of the maximum number of unused segment object lists that can be accommodated; the supplemental threshold may be adjusted to accommodate one quarter or one eighth of the maximum number at high load operation of the metadata server to reduce the number of space application requests that it needs to process when the metadata server load pressure is high.
It can be seen that in this embodiment, the third party development tool needs to communicate with the metadata server only when the number of unused segment objects is insufficient, requesting more unused segment objects. Compared with the interaction with the metadata server in each writing, the design reduces network overhead and saves bandwidth and resource consumption. By creating multiple unused segmented objects in readiness at once and requesting more unused segmented objects from the metadata server when needed, the system can dynamically expand the available space as needed. The design can adapt to the increasing data volume and the requirements of users, and the expandability of the system is maintained.
In some embodiments, the structure of the unused segment object contains the following fields: segment ID, segment status, the corresponding partition object of the segment and the segment ID of the segment to which the segment belongs.
Specifically, a structure of a segmented object is predefined, and the structure represents a segmented space, which contains the following fields:
segment ID: a unique identifier of the segment.
Segment status: may be "unused", "in use", "full of memory", etc.
The partition (existence) object corresponding to the segment.
The segment ID to which the segment belongs: unique identifier of segment (chunk) to which segment belongs
In some embodiments, the segment status includes unused and in-use, and correspondingly, after writing the data in the target segment object into the target real space, further includes:
the target segment object is deleted from the unused segment object list.
In particular, third party development tools may maintain a certain number of unused segmented objects in a local list of unused segmented objects, which results in unused segmented objects that may take up a significant amount of storage space over time. Therefore, after the task of the target segment object is finished and the data is written into the target actual space, the target segment object is timely deleted from the unused segment object list, the occupation of the unused segment object to the storage space is reduced as much as possible, and the new unused segment object is conveniently supplemented in the unused segment object list.
In some embodiments, the method further comprises: and sending a partition creation request to the data storage server, wherein the partition creation request is used for indicating the data storage server to create a new partition according to the partition creation request, and asynchronously reporting a reminding message to the metadata server after the new partition is written with data.
In particular, in the process of writing data to be stored to a data storage server by using no segment object, a third party development tool may have a problem of insufficient partition (extension) on the data storage server. In previous solutions, the metadata server needs to intervene at this time, and after selecting an appropriate data storage server to store a new partition, the third party development tool can continue writing, which undoubtedly increases the complexity of data storage. Therefore, when the situation occurs, the third party development tool directly sends a partition creation request to the data storage server, namely, the third party development tool and the data storage server bypass the metadata server and automatically create and use a new partition, and after the creation of the new partition is completed and data is written into the new partition, the third party development tool asynchronously reports a reminding message to the metadata server to remind the metadata server to correspondingly update the state of the new partition locally, so that the situation of 'get on first and make up later' is realized. Thus, the participation of the metadata server in writing data can be further reduced, and the io path is optimized.
In some embodiments, the metadata server updates the state of the new partition to a normal state after verification and validity check of the alert message is passed.
Specifically, if the third party development tool expects that the message is asynchronously reported to the metadata server after applying for the new partition to the data storage server, and changes the state of the new partition on the metadata server to be a normal state, the specific steps are as follows:
the third party development tool sends a partition creation request to the data storage server, applies for a new partition, and performs a write operation.
After creating a new partition and completing writing, the data storage server asynchronously reports the reminding information to the metadata server through the asynchronous reporting. This asynchronous reporting process may be implemented using some message queues, asynchronous communication, or event driven mechanisms to ensure that the data storage server does not block waiting for the metadata server's response.
A listener or callback function may be implemented on the metadata server for receiving and processing messages reported by the data storage server and performing corresponding status update operations.
When the metadata server processes the reported message, verification and validity check are required to be performed on the message so as to ensure safety and data consistency. Once validated, the metadata server may update the state of the new partition to a normal state.
When the third party development tool writes to the file of the data storage server through the partition ID recorded in the target segment object for the first time, all writable new partition information in the data storage server needs to be acquired from the metadata server, the state is confirmed to the data storage server, and writing can be performed after confirmation is completed.
It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.
Based on the same inventive concept, the embodiment of the application also provides a data distributed storage device. The implementation of the solution provided by the device is similar to the implementation described in the above method, so the specific limitation in one or more embodiments of the data distributed storage device provided below may refer to the limitation of the data distributed storage method hereinabove, and will not be repeated herein.
As shown in fig. 3, an embodiment of the present application provides a data distributed storage device 300, including:
the application module 301 is configured to send a space application request to the metadata server, instruct the metadata server to generate a plurality of unused segment objects at a time according to the space application request, and issue the plurality of unused segment objects to a third party development tool;
a writing module 302, configured to select a target segment object from the local unused segment objects, and write the data to be stored into the target segment object;
the creating module 303 is configured to create, if the target segment object is full, a target actual space on a corresponding segment of the data storage server according to the segment ID recorded in the target segment object;
and the storing module 304 is configured to store the data in the target segment object into the target real space.
In some embodiments, the write module 320 includes:
the adding unit is used for adding a plurality of unused segmented objects issued by the metadata server into a local unused segmented object list;
a selection unit configured to select an unused segment object from the unused segment object list as a target segment object;
correspondingly, the device further comprises:
and the reiteration module is used for sending the space application request to the metadata server again when the number of unused segmented objects in the unused segmented object list is lower than the supplement threshold.
In some embodiments, the structure of the unused segment object contains the following fields: segment ID, segment status, the corresponding partition object of the segment and the segment ID of the segment to which the segment belongs.
In some embodiments, the apparatus further comprises:
and the deleting module is used for deleting the target segmented object from the unused segmented object list.
In some embodiments, an apparatus comprises:
the creating module is used for sending a partition creating request to the data storage server, and is used for indicating the data storage server to create a new partition according to the partition creating request and asynchronously reporting a reminding message to the metadata server after the new partition is written with data.
In some embodiments, the metadata server updates the state of the new partition to a normal state after verification and validity check of the alert message is passed.
The various modules in the data distributed storage device described above may be implemented in whole or in part in software, hardware, and combinations thereof. The above modules may be embedded in hardware or independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In some embodiments, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 4. The computer device includes a processor, a memory, an Input/Output interface (I/O) and a communication interface. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface is connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is for storing unused segment objects. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement the steps in the data distributed storage method described above.
In some embodiments, a computer device is provided, which may be a terminal, and the internal structure of which may be as shown in fig. 5. The computer device includes a processor, a memory, an input/output interface, a communication interface, a display unit, and an input device. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless mode can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement the steps in the data distributed storage method described above. The display unit of the computer device is used for forming a visual picture, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen; the input device of the computer equipment can be a touch layer covered on a display screen, can also be keys, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.
It will be appreciated by those skilled in the art that the structures shown in fig. 4 or 5 are block diagrams of only some of the structures associated with the aspects of the present application and are not intended to limit the computer devices to which the aspects of the present application may be applied, and that a particular computer device may include more or less components than those shown, or may combine some of the components, or have a different arrangement of components.
In some embodiments, a computer device is provided, comprising a memory storing a computer program and a processor implementing the steps of the method embodiments described above when the computer program is executed.
In some embodiments, an internal structural diagram of a computer-readable storage medium is provided as shown in fig. 6, the computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method embodiments described above.
In some embodiments, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the method embodiments described above.
It should be noted that, the user information (including, but not limited to, user equipment information, user personal information, etc.) and the data (including, but not limited to, data for analysis, stored data, presented data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by each party, and the collection, use and processing of the related data are required to comply with the related laws and regulations and standards of the related countries and regions.
Those skilled in the art will appreciate that implementing all or part of the above-described methods may be accomplished by way of a computer program, which may be stored on a non-transitory computer readable storage medium and which, when executed, may comprise the steps of the above-described embodiments of the methods. Any reference to memory, database, or other medium used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the various embodiments provided herein may include at least one of relational databases and non-relational databases. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic units, quantum computing-based data processing logic units, etc., without being limited thereto.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The foregoing examples represent only a few embodiments of the present application, which are described in more detail and are not thereby to be construed as limiting the scope of the present application. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application shall be subject to the appended claims.
Claims (10)
1. A method of data distributed storage, for use with a third party development tool, comprising:
a space application request is sent to a metadata server, and the space application request is used for indicating the metadata server to generate a plurality of unused segmented objects at one time according to the space application request, and issuing the plurality of unused segmented objects to a third party development tool;
selecting a target segment object from the local unused segment objects, and writing data to be stored into the target segment object;
if the target segment object is fully written, creating a target actual space on a corresponding segment of the data storage server according to the segment ID recorded in the target segment object;
and storing the data in the target segment object into the target actual space.
2. The method of claim 1, wherein selecting the target segment object from the local unused segment objects comprises:
adding the plurality of unused segmented objects issued by the metadata server to a local unused segmented object list;
selecting an unused segment object from the unused segment object list as a target segment object;
correspondingly, the method further comprises the steps of:
and when the number of unused segmented objects in the unused segmented object list is lower than a supplement threshold value, resending the space application request to the metadata server.
3. The method of claim 2, wherein the structure of unused segment objects contains the following fields: segment ID, segment status, the corresponding partition object of the segment and the segment ID of the segment to which the segment belongs.
4. A method according to claim 3, wherein the segmentation status includes unused and in-use, and wherein the writing of the data in the target segment object to the target real space is followed by:
deleting the target segment object from the unused segment object list.
5. The method according to claim 1, wherein the method further comprises:
and sending a partition creation request to the data storage server, wherein the partition creation request is used for indicating the data storage server to create a new partition according to the partition creation request, and asynchronously reporting a reminding message to the metadata server after the new partition is written with data.
6. The method of claim 5, wherein the metadata server updates the state of the new partition to a normal state after verification and validity check of the reminder message pass.
7. A data distributed storage device, deployed on a third party development tool, comprising:
the application module is used for sending a space application request to the metadata server, indicating the metadata server to generate a plurality of unused segmented objects at one time according to the space application request, and issuing the plurality of unused segmented objects to a third party development tool;
the writing module is used for selecting a target segmented object from the local unused segmented objects and writing the data to be stored into the target segmented object;
the creating module is used for creating a target actual space on a corresponding block of the data storage server according to the block ID recorded in the target segment object if the target segment object is fully written;
and the storing module is used for storing the data in the target segment object into the target actual space.
8. The apparatus of claim 7, wherein the writing module comprises:
an adding unit, configured to add the plurality of unused segment objects issued by the metadata server to a local unused segment object list;
a selecting unit, configured to select an unused segment object from the unused segment object list as a target segment object;
correspondingly, the device further comprises:
and the reiteration module is used for sending the space application request to the metadata server again when the number of unused segmented objects in the unused segmented object list is lower than a supplement threshold value.
9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 6 when the computer program is executed.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311660187.3A CN117667865A (en) | 2023-12-06 | 2023-12-06 | Data distributed storage method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311660187.3A CN117667865A (en) | 2023-12-06 | 2023-12-06 | Data distributed storage method, device, equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117667865A true CN117667865A (en) | 2024-03-08 |
Family
ID=90076567
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311660187.3A Pending CN117667865A (en) | 2023-12-06 | 2023-12-06 | Data distributed storage method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117667865A (en) |
-
2023
- 2023-12-06 CN CN202311660187.3A patent/CN117667865A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115470156A (en) | RDMA-based memory use method, system, electronic device and storage medium | |
| US11429311B1 (en) | Method and system for managing requests in a distributed system | |
| CN112698793B (en) | Data storage method, device, machine-readable medium and equipment | |
| CN112650729B (en) | Rights management method, system and storage medium of distributed file system | |
| CN114089921A (en) | Power system data storage method and device, computer equipment and storage medium | |
| CN116541336A (en) | Software running method of multi-core chip and coprocessor | |
| CN114924911B (en) | Method, device, equipment and storage medium for backing up effective data of Windows operating system | |
| CN116578410A (en) | Resource management method, device, computer equipment and storage medium | |
| CN117667865A (en) | Data distributed storage method, device, equipment and medium | |
| CN114253481A (en) | Data storage method and device, computer equipment and storage medium | |
| CN114493598A (en) | Computing resource management method, device, computer equipment and storage medium | |
| CN103685359A (en) | Data processing method and device | |
| US11816088B2 (en) | Method and system for managing cross data source data access requests | |
| CN114968092B (en) | Method for dynamically supplying storage space based on QCOW2 technology under container platform and application | |
| CN116756148A (en) | Incremental data acquisition method, device, equipment, storage medium and program product | |
| CN117435129A (en) | Storage cluster expansion method and device, computer equipment and storage medium | |
| CN117271656A (en) | Data synchronization method, device and computer equipment | |
| CN116389244A (en) | Resource allocation correction method, apparatus, device, medium and program product | |
| CN117290354A (en) | Data processing method, device, computer equipment and storage medium | |
| CN116909705A (en) | Batch task processing method and system | |
| CN116775284A (en) | Task processing method, device, computer equipment and storage medium | |
| CN118312310A (en) | Dynamic capacity expansion method, system, terminal and medium for task machine of BIM system | |
| CN116431286A (en) | Virtual machine system, method of controlling virtual machine, computer device, and storage medium | |
| CN118227286A (en) | Task batch processing method and device based on multiparty safety calculation | |
| CN117331625A (en) | Page resource loading method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |