KR102419165B1 - Io redirection methods with cost estimation - Google Patents

Abstract

A distributed storage system node is disclosed. A distributed storage system node may include at least one storage device, which acts as a primary replica for data according to input/output (I/O) requests. The cost analyzer may calculate a local estimated time taken to complete an I/O request at the primary replica, and a remote estimated time taken to complete an I/O request at the secondary replica. The I/O redirector may direct I/O requests to either the primary replica or the secondary replica based on a local estimated time spent and one remote estimated time spent.

Description

IO REDIRECTION METHODS WITH COST ESTIMATION

The present invention relates to a storage device, and more particularly, to improving an input/output (I/O) operation that may be delayed by a primary storage device.

Distributed storage systems such as Ceph use data replication and/or erasure coding to ensure data availability across drive and storage node failures. These distributed storage systems may use solid state drives (SSDs). SSDs have advantages over traditional hard disk drives in that data access is fast and data access does not depend on the location on the drive where the data may be.

SSDs read and write data in units of pages. That is, the entire page is accessed to read some data, and the entire page is written to available pages on the SSD to write some data. But when data is written, that data is written to a blank page. That is, existing data is not overwritten. Therefore, when data is modified on the SSD, the existing page is marked as invalid and a new page is written to the SSD. Therefore, pages of SSDs have one of three states. The three states include an empty state (available state, free), a valid state (a state that stores data, valid), and an invalid state (a state that no longer stores valid data, invalid).

Over time, invalid pages accumulate on the SSD and invalid pages have to be brought to the empty state. However, SSDs erase (erase) data in units of blocks (including several pages) or superblocks (including several blocks). If the SSD waits until all pages in the erase block or superblock become invalid before attempting to delete a block or superblock, then the SSD is full, with no free blocks, and no blocks can be freed. This can be achieved. Therefore, recovering invalid pages may include moving valid pages from one block to another so that the entire block (or super block) can be erased.

Erasing blocks or superblocks is time consuming, which is related to the time required to perform reads and writes. Also, while a block or superblock is being erased, part or the entire SSD of the SSD may not be used. Therefore, managing when SSDs perform garbage collection can be important. For example, if all SSDs in a distributed storage system are garbage collected at the same time, no data requests can be served, and there is nothing better (albeit temporary) than a system that stores data locally and is performing garbage collection. It can be a distributed storage system.

A method is needed to reduce the impact of garbage collection operations in a distributed storage system.

The present invention may provide an apparatus and system for an I/O relocation method.

According to an embodiment of the present invention, a distributed storage system node may include at least one storage device, a cost analyzer, and an I/O relocator. At least one storage device may contain a primary copy of the data. The cost analyzer calculates a local estimated time to complete an input/output (I/O) request on the primary replica and at least one remote estimated time to complete an I/O request on at least one secondary replica of the data. can The I/O redirector may direct the I/O request to one of the primary replica and the at least one secondary replica in response to the local estimated duration and the at least one remote estimated duration.

According to an embodiment of the present invention, input/output (I/O) performance of a distributed storage system may be improved.

1 is a block diagram illustrating a distributed storage system according to an embodiment of the present invention.
FIG. 2 is a block diagram specifically illustrating the storage node of FIG. 1 .
3 is a block diagram specifically illustrating the storage node of FIGS. 1 to 2 .
FIG. 4 is a flow diagram showing the steps a solid state drive (SSD) may be in use of the distributed storage system of FIG. 1 ;
FIG. 5 is a block diagram illustrating the device garbage collection monitor of FIG. 2 receiving free erase block counts from the SSDs of FIGS. 1-2;
FIG. 6 is a block diagram illustrating the device garbage collection monitor of FIG. 2 that selects an SSD for garbage collection based on pre-erase block counts of FIG. 5 from the SSDs of FIGS. 1-2;
FIG. 7 is a block diagram illustrating the device garbage collection monitor of FIG. 2 for estimating a time required to perform garbage collection in the SSD selected in FIG. 6 .
FIG. 8 is a block diagram showing the garbage collection coordinator of FIG. 2 interacting with the monitor of FIG. 1 to perform and schedule garbage collection on the SSD selected in FIG. 6 ;
9A to 9B are block diagrams illustrating input/output (I/O) relocation of FIG. 2 for processing read requests for the SSD selected in FIG. 6 according to embodiments of the present invention.
FIG. 10 is a block diagram illustrating the I/O relocation of FIG. 2 for storing write requests in a logging device for the SSD selected in FIG. 6 .
11 is a block diagram illustrating the I/O resynchronizer of FIG. 2 for processing write requests stored in the logging device of FIG. 10 .
12 is a block diagram illustrating the I/O resynchronizer of FIG. 2 that copies data to the SSD selected in FIG. 6 according to an embodiment of the present invention.
13 is a block diagram specifically illustrating the monitor of FIG. 1 .
14 is a conceptual diagram illustrating an exemplary data map of FIG. 13 stored in the monitor of FIG. 1 .
15A to 15B are flowcharts illustrating a procedure used by the storage node of FIG. 1 for performing garbage collection in the SSD of FIGS. 1 to 2 , according to an embodiment of the present invention.
16A to 16B are flowcharts illustrating a procedure used by the device garbage collection monitor of FIG. 2 for selecting an SSD for garbage collection, according to an embodiment of the present invention.
17 is a flowchart illustrating a procedure for the garbage collection coordinator of FIG. 2 to schedule garbage collection for the SSD selected in FIG. 6 .
18 is a flowchart illustrating a procedure for relocating read requests by the I/O relocation of FIG. 2 according to an embodiment of the present invention.
19 is a flowchart illustrating a procedure for the I/O resynchronizer of FIG. 2 to process logged write requests, according to an embodiment of the present invention.
20 and 21 are flowcharts illustrating a procedure for the monitor of FIG. 1 and the SSD selected in FIG. 6 to handle the time during which garbage collection is performed, according to embodiments of the present invention.
22A to 22B are flowcharts illustrating a procedure for the monitor of FIG. 1 to determine a start time and duration of garbage collection for the SSD selected in FIG. 6 , according to an embodiment of the present invention.
23 is a view showing a client sending an I/O request to the storage node of FIG. 1 , and then relocating the I/O request to another node including a copy of the requested data, according to an embodiment of the present invention. ) is a block diagram showing
24 is a block diagram specifically showing the cost analyzer of FIG. 23 .
25 is a block diagram specifically illustrating the I/O relocation of FIG. 23 .
26 is a block diagram specifically illustrating the local time estimator of FIG. 24 .
FIG. 27 is a block diagram specifically illustrating the remote time estimator of FIG. 24 .
28 and 29 are block diagrams illustrating the local garbage collection time calculator and the local expected garbage collection calculator of FIG. 26 for calculating a local garbage collection time and a local expected garbage collection time.
30 is a block diagram illustrating the queue processing time calculator of FIG. 26 for calculating a queue processing time.
FIG. 31 is a block diagram specifically showing the database of FIG. 24 .
32 is a block diagram specifically illustrating the local prediction analyzer of FIG. 24 .
33 is a block diagram specifically illustrating the local estimated required time calculator of FIG. 26 .
34 is a block diagram specifically showing the communication time calculator of FIG. 27 .
FIG. 35 is a block diagram specifically illustrating the remote processor time calculator of FIG. 27 .
FIG. 36 is a block diagram specifically illustrating the remote prediction analyzer of FIG. 24 .
37 is a block diagram specifically illustrating the remote estimated required time calculator of FIG. 27 .
38 is a block diagram specifically illustrating the I/O relocation of FIG. 25 .
39A to 39B are flowcharts illustrating a procedure for determining where the cost analyzer and I/O relocator of FIG. 23 determine where to send an I/O request, according to an embodiment of the present invention.
40 is a flowchart illustrating a procedure for calculating a local estimated required time by the local estimated required time calculator of FIG. 26 according to an embodiment of the present invention.
41A to 41B are procedures for calculating garbage collection times by the local garbage collection time calculator and local expected garbage collection time calculator of FIG. 26, and the remote garbage collection time calculator of FIG. 27, according to an embodiment of the present invention. is a flow chart showing
42 is a flowchart illustrating a procedure for calculating a queue processing time by the queue processing time calculator of FIG. 26 according to an embodiment of the present invention.
43 is a flowchart illustrating a procedure for estimating a required time required to process an I/O request according to an embodiment of the present invention.
44 is a flowchart illustrating a procedure for using linear regression analysis to determine the weights of FIGS. 26 and 27 , according to an embodiment of the present invention.
45 is a flowchart illustrating a procedure for calculating the remote estimated required time by the remote estimated required time calculator of FIG. 27 according to an embodiment of the present invention.
46 is a flowchart illustrating a procedure for the communication time calculator of FIG. 27 to determine the communication time with the secondary replica, according to an embodiment of the present invention.
47 is a flowchart illustrating a procedure for the remote processor time calculator of FIG. 27 to determine a remote processor time, according to an embodiment of the present invention.

Reference is made to embodiments of the technical idea of the present invention, examples of which are shown in the accompanying drawings. In the following detailed description, various specific details are provided in order to provide a sufficient understanding of the technical spirit of the present invention. However, those skilled in the art may implement the technical spirit of the present invention without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

Although terms such as first, second, etc. are used herein to describe various elements, these elements are not limited by these terms. These terms are only used to distinguish one element from another. For example, a first module may be referred to as a second module without departing from the scope of the inventive concept. Likewise, the second module may be referred to as the first module.

Terms used in the description of the technical spirit of the present invention are used only for the purpose of describing specific embodiments, and are not intended to limit the technical spirit of the present invention. As used in the description of the present invention and in the appended claims, singular expressions are intended to include plural expressions as well unless the context clearly dictates otherwise. It is intended that the term “and/or” includes any and all possible combinations of one or more associated items. The terms “comprises” and/or “comprising,” when used in the specification, specify the presence of recited properties, integers, steps, operations, elements, and/or components, one or It does not exclude the presence or addition of many other properties, integers, steps, operations, elements, components, and/or groups thereof. Elements and characteristics in the drawings are not necessarily to scale.

1 is a block diagram illustrating a distributed storage system according to an embodiment of the present invention. In FIG. 1 , clients 105 , 110 , and 115 are shown in communication with a network 120 . Clients 105, 110, and 115 may be any desired types of devices, including desktop computers, notebook computers, tablet computers, servers, smartphones, and the like. Also, although Figure 1 shows three clients 105, 110, and 115, embodiments of the present invention may support any number of clients. Clock synchronization may be maintained across various storage nodes of the storage system using services such as Network Time Protocol and the like.

The network 120 may be any of a variety of networks including a global network such as a local area network (LAN), a wide area network (WAN), and the Internet. Network 120 may also include a combination of several different networks. For example, the network 120 may include a plurality of local area networks capable of communicating with each other through a global network using a virtual private network (VPN) to secure communication.

Storage nodes 125 , 130 , and 135 are also coupled to network 120 . Each storage node 125 , 130 , and 135 provides storage for the distributed network. Each storage node 125 , 130 , and 135 includes flash drives (also referred to as solid state drives or SSDs) 140 , 145 in storage node 125 , and flash drives 150 in storage node 130 . , 155 ), and various storage devices such as flash drives 160 , 165 in storage node 135 . Although FIG. 1 shows three storage nodes 125 , 130 , and 135 , embodiments of the present invention may support any number of storage nodes. Also, although FIG. 1 shows each storage node 125, 130, and 135 supporting two flash drives, embodiments of the present invention may It can support any number of flash drives. Storage nodes 125 , 130 , and 135 may include other types of storage devices, such as traditional hard disk drives that may or may not benefit from coordinated garbage collection. For example, traditional hard disk drives sometimes require defragmentation to merge parts of scattered files. Defragmentation can be a time consuming operation and can also benefit from coordination, such as garbage collection on flash drives. Storage nodes 125 , 130 , and 135 may also include variants of SSDs, such as network attached SSDs and Ethernet SSDs. Storage nodes 125 , 130 , and 135 are further described with reference to FIGS. 2-3 and 5-12 below.

Monitors (also referred to as monitor nodes) 170 , 175 , and 180 are also coupled to network 120 . Monitors 170 , 175 , and 180 are responsible for tracking cluster configuration and notifying entities of changes in cluster configuration. Examples of such changes may include addition or removal of storage nodes, changes in availability of storage on storage nodes (such as addition or removal of a flash drive), and the like. Note that "changes" in this context are not limited to intentional actions to change distributed storage systems. For example, if the network connection is lost, the storage node 125 is removed from the distributed storage system, even though the storage node is still up and trying to communicate with the rest of the distributed system, and its operation monitors the cluster configuration ( 170, 175, and 180). Monitors 170 , 175 , and 180 are further described with reference to FIGS. 4 , 8 , and 13 below.

Although the remainder of the description below focuses on SSDs, embodiments of the present invention may be applied to any storage devices that perform garbage collection in a manner similar to SSDs. Any reference to SSD below is also intended to include other storage devices that perform garbage collection as well.

FIG. 2 is a block diagram specifically illustrating the storage node 125 of FIG. 1 . In Fig. 2, the storage node 125 is specifically indicated. Storage nodes 130 , 135 may be similar to storage node 125 of FIG. 2 . The storage node 125 includes a device garbage collection monitor 205 , a garbage collection coordinator 210 , an input/output (I/O) relocator 215 , an I/O resynchronizer 220 , and four flash drives. (140, 145, 225, and 230). The device garbage collection monitor 205 may determine which flash drives should perform garbage collection, and may estimate the time it takes for the flash drives to perform garbage collection. The garbage collection coordinator 210 may communicate with one or more monitors 170 , 175 , and 180 of FIG. 1 to schedule garbage collection for the selected flash drive, and also when to perform garbage collection on the selected flash drive. and how long it will hold garbage collection. The I/O relocation 215 may relocate read and write requests directed to the selected flash driver while the flash drive performs garbage collection. Also, the I/O resynchronizer 220 may bring the flash drive up to date with respect to data changes after garbage collection is complete.

3 is a block diagram specifically illustrating the storage node 125 of FIGS. 1 to 2 . 3 , generally, storage node 125 includes one or more processors, including memory controller 310 and clock 315 , which may be used to coordinate operations of components of storage node 125 . 305) may be included. Processors 305 may be coupled with memory 320 , which may include, as examples, random access memory (RAM), read-only memory (ROM), or other stateful medium. . Processors 305 may also be coupled to storage devices 140 and 145 , and a network connector 325 , which may be, for example, an Ethernet connector. Processors 305 may also be coupled to a bus 330 , which, among other components, includes a user interface 335 and input/output that may be managed using an input/output engine 340 . may be attached to the interface ports. Another representation of a storage node could be a single package or ASIC, as shown in FIG. 3 , and all discrete components that are directly connected to the network 120 of FIG. 1 .

FIG. 4 is a flow chart showing the steps in which the distributed storage system of FIG. 1 may have a solid state drive (SSD) in use; In FIG. 4 , SSDs start at configuration step 405 . In the configuration step 405 , the SSDs may be configured to perform garbage collection by command only from the storage node of FIGS. 1-2 . As indicated by the dashed line, the configuration step 405 is optional. Some SSDs do not support configuration, or even though SSDs may be configured, a distributed storage system may choose not to configure SSDs. Then, in a normal step 410 , the SSDs operate normally in response to the forwarded read and write requests.

When the SSD needs to perform garbage collection, the SSD may enter a preparation phase 415 . The preparation step 415 may include determining when the SSD will perform garbage collection and how much time the SSD should have to perform garbage collection. Note that in the preparation phase 415 , the SSD can still process read and write requests normally. Thereafter, at an appropriate time, the SSD may enter the display step 420 . At mark step 420 , monitors 170 , 175 , and 180 of FIG. 1 may mark the SSD as performing garbage collection, and thus the SSD may be unavailable. The SSD may then enter a garbage collection phase 425 , where the SSD may perform a garbage collection.

When the SSD completes garbage collection, or when the time allotted for garbage collection has expired, the SSD may enter the resynchronization phase 430 . In the resynchronization step 430 , the SSD may be updated with respect to data changes affecting the SSD. After the resynchronization step 430 is complete, the SSD may enter an indication removal step 435 , where the monitors 170 , 175 , and 180 of FIG. 1 may again indicate the SSD as available ( That is, the operations in the display step 420 are reversed). Finally, the SSD returns to the normal step 410 and may normally process read and write requests.

FIG. 5 is a block diagram illustrating the device garbage collection monitor of FIG. 2 receiving free erase (erase) block counts from the SSDs of FIGS. 1-2; In FIG. 5 , the device garbage collection monitor 205 is shown interacting with the flash drives 140 , 145 , 225 , and 230 of the storage node 125 of FIG. 1 . In general, the device garbage collection monitor 205 only interacts with storage devices on the storage node that contains the device garbage collection monitor 205 . However, embodiments of the present invention may have a device garbage collection monitor 205 that interacts with the storage devices of other storage nodes.

Device garbage collection monitor 205 may periodically receive free erase block counts (FEBs) 505 , 510 , 515 , and 520 from flash drives 140 , 145 , 225 , 230 , respectively. The pre-erase block counts 505 , 510 , 515 , and 520 may indicate the percentage or number of currently empty blocks (or super blocks) of the flash drives 140 , 145 , 225 , and 230 respectively. have. Regarding the total number of usable blocks of the SSD, the pre-erase block count can be a good indicator of how full the SSD is. As the pre-erase block count decreases, the SSD is filling up, and garbage collection may be needed to increase the number of pre-erase blocks.

In some embodiments of the present invention, flash drives 140 , 145 , 225 , and 230 automatically send pre-erase block counts 505 , 510 , 515 , and 520 to device garbage collection monitor 205 . In other embodiments of the present invention, device garbage collection monitor 205 informs flash drives 140, 145, 225, and 230 when flash drives 140, 145, 225, and 230 have their pre-erase block count. You can ask if you would like to know the fields 505 , 510 , 515 , and 520 . These queries are indicated by polls 525 530 , 535 , and 540 respectively. All embodiments of the present invention have a device garbage collection monitor 205 that obtains information of pre-erase block counts 505, 510, 515, and 520 in flash drives 140, 150, 225, and 230. poles 525 , 530 , 535 , and 540 are indicated by dashed lines.

Flash drives 140 , 145 , 225 , and 230 may return more than just pre-erase block counts 505 , 510 , 515 , and 520 to the device garbage collection monitor 205 . For example, flash drives 140 , 145 , 225 , and 230 may indicate to the device garbage collection monitor that it needs to perform static wear leveling. Simply put, data cells in an SSD can perform a lot of write and erase operations before the data cells start failing (misbehaving). The manufacturer of an SSD knows how many write and erase operations can occur on average in data cells. SSDs may use static wear leveling to try to keep the number of write and erase operations fairly evenly across all data cells, preferably avoiding initial errors of the SSD due to overusing a small number of data cells. can make it

6 is a device garbage collection monitor of FIG. 2 that selects an SSD for garbage collection based on pre-erase block counts 505 , 510 , 515 , and 520 of FIG. 5 received from the SSDs of FIGS. 1 and 2 . It is a block diagram showing (205). 6 , device garbage collection monitor 205 may use comparator 605 to compare pre-erase block counts 505 , 510 , 505 , and 520 with pre-erase block threshold 610 . If any one of the pre-erase block counts 505, 510, 505, and 520 is less than the pre-erase block threshold 610, then the SSD providing that pre-erase block count is the selected SSD for garbage collection ( 615) can be

6 shows a single pre-erase block threshold 610 , embodiments of the present invention may support multiple pre-erase block thresholds 610 . That is, depending on the flash drive in question, the comparator 605 may compare the pre-erase block counts 505 , 510 , 505 , and 520 with different pre-erase block thresholds 610 . In this way, embodiments of the present invention may recognize SSDs that may have different block sizes and counts, and thus may have different thresholds indicating how full the flash drive is.

In some embodiments of the present invention, the pre-erase block threshold 610 may be a fixed value. For example, consider an SSD with a total capacity of 512 GB and a block size of 2048 KB. This SSD has 250,000 blocks. Such an SSD may have a pre-erase block threshold 610 set to 50,000. On the other hand, an SSD with a total capacity of 256 GB and a block size of 512 KB will have 500,000 blocks and may have a pre-erase block threshold 610 set to 100,000.

In other embodiments of the present invention, the pre-erase block threshold 610 may be a percentage, such as 20%. That is, when the SSD has a pre-erase block count that is less than or equal to 20% of the total number of blocks, the SSD needs to perform garbage collection. Setting the pre-erase block threshold 610 to 20% (not a fixed value) is the two example SSDs described above, requiring different pre-erase block thresholds when using a fixed number of blocks. Note that it can contain all of them.

FIG. 7 is a block diagram illustrating the device garbage collection monitor 205 of FIG. 2 for estimating the time required to perform garbage collection on the selected SSD 615 of FIG. 6 . In FIG. 7 , the device garbage collection monitor 205 may include a time estimator 705 . The time estimator 705 may estimate a time required to perform garbage collection in the selected SSD 615 of FIG. 6 .

The time estimator 705 may use various data to estimate a time required to perform garbage collection in the selected SSD 615 of FIG. 6 . This data includes the erase cycle time 710 (the time required to erase a block of the selected SSD 615 in FIG. 6), the preceding time required 715 (the SSD selected at the previous time in FIG. 6 that performed garbage collection). time required by the selected SSD 615 to perform garbage collection at 615 ), and the SSD capacity 720 (total capacity of the selected SSD 615 in FIG. 6 ). If the number of blocks to be erased is known, the time estimator 705 may also use the number of blocks to be erased in the selected SSD 615 of FIG. 6 . This data provides a reasonable (if not perfect) estimate of how long the selected SSD 615 of FIG. 6 will perform garbage collection.

8 is a block diagram illustrating the garbage collection coordinator 210 of FIG. 2 interacting with the monitor 170 of FIG. 1 to schedule and perform garbage collection on the SSD 615 selected in FIG. 6 . In FIG. 8 , the garbage collection coordinator 210 determines the identifier 805 of the selected SSD 615 of FIG. 6 with an estimated time 725 required to perform garbage collection on the selected SSD 615 of FIG. 6 and Together, it can be transmitted to the monitor 170 . The garbage collection coordinator 210 may send to the monitor 170 a scheduled start time 810 , which may be selected by the garbage collection coordinator 210 . The monitor 170 may then use this information to learn about garbage collection on the selected SSD 615 of FIG. 6 . This exchange of information is denoted exchange 815 .

There are different models of how monitor 170 may operate. In one model, called GC with Acknowledgment, monitor 170 (which can be coordinated with other monitors in the distributed storage system) determines when each SSD performs garbage collection, and how much. You can determine how long an SSD spends time on garbage collection. In this model, monitor 170 tells garbage collection coordinator 210 when the selected SSD 615 of FIG. 6 can perform garbage collection and how long the selected SSD 615 of FIG. 6 should perform garbage collection. The garbage collection coordinator 210 does not instruct the selected SSD 615 to start garbage collection until it is informed as to whether to do so. That is, the selected SSD 615 of FIG. 6 remains in the normal stage 410 of FIG. 4 until the monitor 170 notifies the garbage collection coordinator 210 about when to perform garbage collection.

In embodiments of the present invention using responsive garbage collection, the scheduled start time 810 and the estimated time 725 selected by the garbage collection coordinator 210 are not bound (bound). Only the scheduled start time 810 and duration 820 specified by the monitor 170 are used. The information sent by the garbage collection coordinator 210 is merely a suggestion to the monitor 170 .

In embodiments of the invention that use responsive garbage collection, monitor 170 (which is coordinating with other monitors in the distributed storage system) may To minimize the impact, you can schedule each SSD that needs garbage collection. For example, when two different SSDs each want to perform garbage collection, the monitor 170 prioritizes which SSD can perform garbage collection first, and causes the other SSD to wait. In this way, only one SSD performs garbage collection at any time, reducing the likelihood that a data request may not be served by either SSD.

In other embodiments of the invention, monitor 170 may operate on a model called unresponsive garbage collection. In this model, monitor 170 can track when SSDs perform garbage collection, but monitor 170 responds to scheduled start time 810 and estimated time 725 selected by garbage collection coordinator 210 . Otherwise, the scheduled start time 810 and the estimated time 725 selected by the garbage collection coordinator 210 are not changed. In embodiments using non-responsive garbage collection, a plurality of SSDs may concurrently perform garbage collection. However, if the data redundancy level in the distributed storage system is sufficient, the possibility that data requests will be delayed until the SSD completes garbage collection is minimal. For example, if the distributed storage system includes three copies of each unit of data, the probability that all three copies will be unavailable when requested by the client 105 of FIG. 1 may be small enough to be acceptable.

There is a mathematical relationship between the number of copies of each unit of data and the likelihood that no copies are available at any time (for example, how often garbage collection occurs on an SSD or how long it takes to be garbage collected). with variables). Given a desired reliability (ie, if at least one copy of each unit data is available at any time), the number of copies of each unit data may be calculated to provide that desired reliability.

Irrespective of whether responsive or non-responsive garbage collection is used, the garbage collection coordinator 210 eventually determines that garbage collection is may inform monitor 170 that it is starting on 140 . The garbage collection coordinator 210 may instruct the flash drive 140 to begin garbage collection, as indicated by the garbage collection start instruction 830 . Eventually, after the duration 820 has elapsed, the garbage collection coordinator 210 may instruct the flash drive 140 to end garbage collection, as indicated by the garbage collection end instruction 835 . Finally, garbage collection coordinator 210 may inform monitor 170 that garbage collection has completed on flash drive 140 , as indicated by garbage collection complete message 840 .

Note that embodiments of the present invention do not require all the messages of FIG. 8 . As described above, in embodiments of the present invention using no-response garbage collection, monitor 170 does not send any response to garbage collection coordinator 210 . Also, since the monitor 170 knows when garbage collection is scheduled to start and end, the garbage collection coordinator 210 may skip sending messages 825 and 840 to the monitor 170 . Also, instruction 835 is useful only when flash drive 140 allows garbage collection to abort. If garbage collection is not aborted on flash drive 140 , then flash drive 140 will not process instruction 835 . However, if flash drive 140 does not allow, via instruction 835 , to stop garbage collection, then message 840 to inform monitor 170 when flash drive 140 has completed garbage collection. may be needed For example, consider a scenario where flash drive 140 does not allow garbage collection to be interrupted, but duration 820 is shorter than the time it takes for garbage collection to be performed on flash drive 140 . If, after the duration 820 has passed, the monitor 170 assumes that the flash drive 140 has completed garbage collection, the monitor 170 displays the flash drive 140 when the flash memory 140 is not available. ) is available, which may result in data requests being delayed while multiple SSDs are performing garbage collection at the same time.

9A to 9B are block diagrams illustrating the input/output (I/O) relocator 215 of FIG. 2 for processing read requests for the SSD 615 selected in FIG. 6 , according to embodiments of the present invention. they are In some embodiments of the invention of FIG. 9A , the I/O relocator 215 may receive a read request 905 that is directed to the flash drive 140 . However, if the flash drive 140 is performing garbage collection, allowing the read request 905 to be delivered to the flash drive 140 may result in delayed return of the requested data. Therefore, I/O relocator 215 may cancel forwarding of read request 905 to flash drive 140 (as indicated by deleted arrow 910 ), and instead I/O relocator ( 215 may redirect read request 905 to flash drive 145 (as indicated by arrow 915 ). I/O relocator 215 maps 920 that can be stored locally or accessed from monitor 170 of FIG. 1 to determine which storage devices in the distributed storage system store different copies of the requested data. can be used I/O relocator 215 may then relocate read request 905 to an appropriate device, such as flash drive 415 . Flash drive 145 can then access data 925 and return data 925 directly to the requesting client.

9B , in other embodiments of the present invention, I/O relocator 215 intercepts (intercepts) read request 905 directed to flash drive 140, which is undergoing garbage collection. can However, instead of relocating the read request 905 to the flash drive 145 , the I/O relocator 215 may itself request data from the flash drive 145 , at the exchange 930 . Flash drive 145 may then return data 925 to I/O relocator 215 in exchange 930 , which then returns data 925 to the requesting client in message 935 . can do. 9B , the requesting client may not receive data from an unexpected source, which may confuse the requesting client if the requesting client is not prepared for this possibility.

As with the read request 905 of FIGS. 9A to 9B , write requests designated to be transferred to the SSD performing garbage collection may be delayed as a result. FIG. 10 is a block diagram illustrating the I/O relocation 215 of FIG. 2 storing write requests to a logging device for the SSD 615 selected in FIG. 6 to avoid these delays. Instead of forwarding the write request 1005 to the flash drive 145 , the I/O relocation 215 may intercept the write request 1005 (as indicated by the deleted arrow 1010 ). I/O relocator 215 may then store write request 1005 to logging device 1015 as indicated by arrow 1020 . The logging device 1015 may be located anywhere within the flash drive 140 , local to the storage node 125 of FIG. 1 , or wherever desired.

11 is a block diagram illustrating the I/O resynchronizer 220 of FIG. 2 for processing write requests stored in the logging device 1015 of FIG. 10 . Once garbage collection is complete on the selected SSD 615 of FIG. 6 , the I/O resynchronizer 220 can access the write request 1005 from the logging device 1015 , as indicated by arrow 1105 . have. I/O resynchronizer 220 may then perform a write request from flash drive 140 , as indicated by arrow 1110 . The write request 1005 may then be deleted from the logging device 1015 , as indicated by delete 1115 .

Although logging device 1015 provides a simple way to ensure that flash drive 140 is currently in progress with respect to data writes, logging device 1015 is the only means for resynchronizing flash drive 140 . not. Another possibility stores information about which pages, blocks, or super blocks of flash drive 140 were to be changed by write requests that arrived while flash drive 140 was garbage collected. may be doing The I/O resynchronizer 220 then accesses the updated data from the replicated copies of these pages/blocks/superblocks on other SSDs or other storage devices, and flashes the updated data. It can be written to the drive 140 . 12 shows this form of resynchronization.

FIG. 12 is a block diagram illustrating the I/O resynchronizer 220 of FIG. 2 that copies data to the SSD 615 selected in FIG. 6 according to embodiments of the present invention. 12 , I/O resynchronizer 220 may request updated data from flash drive 145 and receive data 925 from flash drive 145 at access exchange 1205 . As described above with reference to FIGS. 9A-9B , I/O resynchronizer 220 , possibly using map 920 , identifies which SSDs are storing updated data, and requests data accordingly. will be able Flash drive 145 will not necessarily store all updated data. The I/O resynchronizer 220 may then provide the data 925 to the flash drive 140 in a duplicate message 1210 .

13 is a block diagram specifically illustrating the monitor 170 of FIG. 1 . In FIG. 13 , monitor 170 may include storage 1305 , which may include map 1310 and waiting list 1315 . The monitor 170 may also include a receiver 1330 , a transmitter 1325 , and a scheduler 1320 . The map 1310 may store information regarding where data is stored in various storage devices of a distributed storage system, along with which devices are available and unavailable. Map 1310 is further described with reference to FIG. 14 below. As described above with reference to FIG. 8 , the waiting list 1315 may store information about SSDs that intend to perform garbage collection, but are currently delayed while other SSDs perform garbage collection. The map updater 1335 may update the map 1310 as data storage throughout the distributed storage system changes. Receiver 1330 and transmitter 1325 receive information from other devices, such as garbage collection coordinator 210 of FIG. 2, and send information to other devices, such as garbage collection coordinator 210 of FIG. can be used for The scheduler 1320 may schedule a garbage collection request for the selected SSD 615 of FIG. 6 , and selects a time and duration for garbage collection of the SSD.

FIG. 14 is a conceptual diagram illustrating an exemplary data map 1310 of FIG. 13 stored in the monitor 170 of FIG. 1 . In FIG. 14 , a map 1310 may include indicators of various unit data 1403 , 1406 , 1409 , and 1412 . For example, unit data 1403 identifies block 0 of file A, unit data 1406 identifies block 1 of file A, unit data 1409 identifies block 2 of file A, and unit Data 1412 identifies block 0 of file B. Each unit data may include a count, such as counts 1415 , 1418 , 1421 , and 1424 , the count indicating how many copies of each unit data are there. For example, counts 1415 , 1418 , and 1421 indicate that there are three copies of unit data 1403 , 1406 , and 1409 respectively, and count 1424 is a copy of unit data 1412 . This indicates that there are two.

The map 1310 may also include a location where various unit data can be found. These are indicated by positions 1427 , 1430 , 1433 , 1436 , 1439 , 1442 , 1445 , 1448 , 1451 , 1454 , and 1457 . Location 1460 is available if unit data 1412 eventually has a third copy, but since there are only two copies of unit data 1412 in the distributed storage system, location 1460 is currently empty.

Although FIG. 14 shows a map 1310 including four unit data and up to three copies for each unit data, embodiments of the present invention include any number of unit data, and each unit data. Any number of copies can be supported for Thus, the map 1310 could theoretically contain millions (or more) of unit data, and could have six copies of each unit data throughout the distributed storage system.

15A to 15B are flowcharts illustrating a procedure used by the storage node 125 of FIG. 1 to perform garbage collection on the SSD of FIGS. 1 to 2 , according to an embodiment of the present invention. At block 1505 of FIG. 15A , the device garbage collection monitor 205 of FIG. 2 may select an SSD, such as flash drive 140 of FIG. 1 , to perform garbage collection. At block 1510 , the device garbage collection monitor 205 of FIG. 2 may determine an estimated time required for the flash drive 140 of FIG. 1 to perform garbage collection. Among the factors that the device garbage collection monitor 205 of FIG. 2 may consider are the erase cycle time required for the flash drive 140 , the time required for a preceding garbage collection event by the flash drive 140 of FIG. 1 , and FIG. 1 . There is a usable capacity of the flash drive 140 of. At block 1515 , the garbage collection coordinator 210 of FIG. 2 may determine a scheduled start time for the flash drive 140 of FIG. 1 to perform garbage collection. At block 1520 , at the scheduled start time, garbage collection coordinator 210 of FIG. 2 may instruct flash drive 140 of FIG. 1 to begin garbage collection.

At block 1525 , while flash drive 140 of FIG. 1 performs garbage collection, I/O relocation 215 of FIG. 2 redirects read requests 905 away from flash drive 140 of FIG. 1 . can be rearranged At block 1530 , while flash drive 140 of FIG. 1 is performing garbage collection, I/O relocator 215 sends relocation write requests 1005 of FIG. 10 from flash drive 140 of FIG. It may be relocated separately (to a logging device for storing write requests 1005 of FIG. 10 or another storage device for performing write requests 1005 of FIG. 10 ). At block 1535 , storage node 125 of FIG. 1 may wait until flash drive 140 of FIG. 1 completes garbage collection. At block 1540 , garbage collection coordinator 210 of FIG. 2 may instruct flash drive 140 of FIG. 1 to terminate garbage collection. As described above with reference to FIG. 8 , block 1540 may be omitted if flash drive 140 is not interrupted during garbage collection.

At block 1545 ( FIG. 15B ), I/O resynchronizer 220 of FIG. 2 may determine (at block 1530 ) whether there are relocated write requests 1005 . If there are relocated write requests 1005, then at block 1550 the I/O resynchronizer 220 of FIG. 2 accesses and plays the write request 1005 of FIG. 10 from the logging device 1015 of FIG. and at block 1555 , the I/O resynchronizer 220 of FIG. 2 may delete the write request 1005 of FIG. 10 from the logging device 1015 of FIG. 10 . As described above with reference to FIG. 12 , replaying the write request 1005 of FIG. 10 at block 1550 is more important than actually replaying the original write request 1005 of FIG. 1 , the flash drive 145 of FIG. 1 . ) may include accessing updated data from Therefore, instead, at block 1560 I/O resynchronizer 220 of FIG. 2 is replaced by write request 1005 of FIG. 10 from another storage device that has processed write request 1005 of FIG. Data can be copied. Control may then return to block 1545 to further check write requests 1005 of FIG. 10 in logging device 1015 of FIG. 10 .

16A to 16B are flowcharts illustrating a procedure used by the device garbage collection monitor of FIG. 2 for selecting an SSD for garbage collection, according to an embodiment of the present invention. In FIG. 16A , at block 1605 , the device garbage collection monitor 205 of FIG. 2 polls the pre-erase block counts from the flash drives 140 , 145 , 225 , and 230 of FIGS. 1-2 . can be investigated). At block 1610, the device garbage collection monitor 205 of FIG. 1 performs the pre-erase block counts 505, 510, 515, and 520).

At block 1615 ( FIG. 16B ), the device garbage collection monitor 205 of FIG. 2 may determine if there are remaining free erase block counts to process. If there are remaining pre-erase block counts to process, then at block 1620, the device garbage collection monitor 205 of FIG. 2 may select one of the pre-erase block counts. At block 1625 , the device garbage collection monitor 205 of FIG. 2 may determine whether the selected pre-erase block count is less than the pre-erase block threshold 610 of FIG. 6 . If the selected pre-erase count is less than the pre-erase block threshold 610 of FIG. 6, then at block 1630, the device garbage collection monitor 205 of FIG. 2 may know that the corresponding SSD is eligible for garbage collection. have. Control may then return to block 1615 to check if there are more pre-erase block counts to process.

At block 1615 , if there are no remaining pre-erase block counts to process, then at block 1635 the device garbage collection monitor 205 of FIG. 2 may select one of the SSDs eligible for garbage collection. If there is only one SSD eligible for garbage collection, then that SSD can be selected. If there is more than one eligible SSD, then device garbage collection monitor 205 of FIG. 2 may select one of the eligible SSDs using any desired algorithm. Possible algorithms that may be used include selecting an eligible SSD with the lowest pre-erase block count or the smallest percentage of pre-erase blocks, or randomly selecting an eligible SSD. The device garbage collection monitor 205 of FIG. 2 may also select one or more SSDs eligible for garbage collection. Since the monitor 170 of FIG. 1 can schedule all SSDs eligible for garbage collection in a way that avoids multiple SSDs performing garbage collection at the same time, selecting one or more SSDs eligible for garbage collection is This is particularly useful where the monitor 170 uses responsive garbage collection.

17 is a flowchart illustrating a procedure for the garbage collection coordinator 210 of FIG. 2 to schedule garbage collection for the SSD 615 selected in FIG. 6 . In FIG. 17 , at block 1705 , the garbage collection coordinator 210 of FIG. 2 may select a scheduled start time 810 for the selected SSD 615 of FIG. 6 . As described above with reference to FIG. 8, the scheduled start time 810 selected by the garbage collection coordinator 210 of FIG. 2 does not need to be bound (constrained), and the other scheduled start time 810 of FIG. ) may be assigned by the monitor 170 of FIG. 1 . At block 1710 , the garbage collection coordinator 210 of FIG. 2 may notify the monitor 170 of FIG. 1 that the selected SSD 615 of FIG. 6 needs to perform garbage collection. At block 1715 , the garbage collection coordinator 210 of FIG. 2 estimates the selected start time 810 of FIG. 8 and the estimate of FIG. 7 as needed by the selected SSD 615 of FIG. 6 to perform garbage collection. Time 725 may be communicated to monitor 170 of FIG. 1 .

Here, the flowchart may be divided according to whether the embodiments of the present invention use responsive garbage collection or non-responsive garbage collection. In embodiments of the present invention that use unresponsive garbage collection, the garbage collection coordinator 210 of FIG. 2 at block 1720 determines the schedule of FIG. 8 for the selected SSD 615 of FIG. 6 to perform garbage collection. The selected SSD 615 of FIG. 6 may be instructed to perform garbage collection at the scheduled start time 810 of FIG. 8 using the estimated start time 810 and the estimated time 725 of FIG. 7 .

In embodiments of the present invention that use responsive garbage collection, at block 1725 the garbage collection coordinator 210 of FIG. 2 receives the scheduled start time 810 of FIG. 8 from the monitor 170 of FIG. may receive the duration 820 of . Thereafter, at block 1730 the garbage collection coordinator 210 compares the scheduled start time 810 of FIG. 8 and the duration 820 of FIG. 8 received from the monitor 170 of FIG. 1 to the selected SSD of FIG. (615) can be used to schedule when to perform garbage collection.

18 is a flowchart illustrating a procedure for the I/O relocation 215 of FIG. 2 to relocate the read requests 905 of FIG. 9 according to an embodiment of the present invention. In FIG. 18 , I/O relocation 215 of FIG. 2 at block 1805 may identify the flash drive 145 of FIG. 1 as storing a duplicated copy of the requested data. As described above with reference to FIGS. 9A-9B , the I/O relocation 215 of FIG. 2 is used to determine whether flash drive 145 of FIG. 1 stores a duplicated copy of the requested data. The map 920 of FIGS. 9A-9B may be used, and the map 920 of FIGS. 9A-9B is stored locally on the storage node 125 of FIG. 1 on the monitor 170 of FIG. 1 or stored anywhere desired. can be

Once the I/O relocator 215 of FIG. 2 has determined that the flash drive 145 of FIG. 1 is storing a duplicated copy of the requested data, other embodiments of the present invention may proceed with other methods. have. In some embodiments of the present invention, I/O relocation 215 of FIG. 2 at block 1810 may simply relocate read request 905 of FIGS. 9A-9B to flash drive 145 of FIG. 1 . In this case, the flash drive 145 of FIG. 1 may directly provide the data 905 of FIGS. 9A to 9B to the requesting client. In other embodiments of the invention, at block 1815 I/O relocation 215 of FIG. 2 requests data from flash drive 145 of FIG. 1 , and at block 1820 I/O of FIG. 2 . The relocator 215 may provide the data 925 of FIGS. 9A-9B to the requesting client.

19 is a flowchart illustrating a procedure for the I/O resynchronizer 220 of FIG. 2 to process the logged write requests 1005 of FIG. 10 according to an embodiment of the present invention. How the I/O resynchronizer 220 of FIG. 2 handles the logged write requests 1005 of FIG. 10 depends on how the logging device 1015 of FIG. 10 is used. If the logging device 1015 of FIG. 10 stores the write requests 1005 of FIG. 10 as initially generated by the requesting client, then in block 1905, the I/O resynchronizer 220 of FIG. 2 is as if You can simply replay the write requests 1005 of FIG. 10 as you just received.

However, the logging device 1015 of FIG. 10 may store an indicator of which pages, blocks, or superblocks are affected by the write requests 1005 of FIG. 10 . In that case, at block 1910 I/O resynchronizer 220 of FIG. 2 may identify flash drive 145 of FIG. 1 as storing a duplicate copy of the updated data. As described above with reference to FIG. 12, the I/O resynchronizer 220 of FIG. 2 uses the map of FIG. 12 ( 920), and the map 920 of FIG. 12 may be stored locally on the monitor 170 of FIG. 1, at the storage node 125 of FIG. 1, or wherever desired.

Once the I/O resynchronizer 220 of FIG. 2 has identified that the flash drive 145 stores a duplicate copy of the updated data, at block 1915 the I/O resynchronizer 220 of FIG. The updated data can be accessed from the flash drive 145 of FIG. 1 . Thereafter, at block 1920 , the I/O resynchronizer 220 of FIG. 2 may instruct the selected SSD 615 of FIG. 6 to write the updated data.

20 and 21 are flowcharts illustrating a procedure for the monitor 170 of FIG. 1 to handle the time when the SSD 615 selected in FIG. 6 performs garbage collection, according to embodiments of the present invention. 20, at block 2005 the monitor 170 of FIG. 1 may receive a notification from the garbage collection coordinator 210 of FIG. 1 that the selected SSD 615 of FIG. 6 needs to perform a garbage collection. have. This notification may include the estimated time 725 of FIG. 7 and the scheduled start time 810 of FIG. 8, which are needed by the selected SSD 615 of FIG. 6 to perform garbage collection. At block 2010, the monitor 170 of FIG. 1 may select a scheduled start time for the selected SSD 615 of FIG. 6 to perform garbage collection, and at block 2015 the monitor 170 of FIG. 170 may select a duration for the selected SSD 615 of FIG. 6 to perform garbage collection. At block 2020 , the monitor 170 of FIG. 1 may inform the garbage collection coordinator 210 of FIG. 2 of the scheduled start time 810 and duration 820 of FIG. 8 .

At block 2025, when the scheduled start time 810 of FIG. 8 for garbage collection arrives, the monitor 170 of FIG. 1 reflects that the selected SSD 615 of FIG. 6 is now unavailable. Data of the map 1310 of FIG. 13 of the distributed storage system may be updated. The monitor 170 of FIG. 1 may know that garbage collection has started in the selected SSD 615 of FIG. 6 through a notification from the garbage collection coordinator 210 of FIG. 2 . At block 2030 , when garbage collection is complete, the monitor 170 of FIG. 1 displays the map 1310 of FIG. 13 of the distributed storage system to reflect that the selected SSD 615 of FIG. 6 is now available again. ) can be updated. The monitor 170 of FIG. 1 may know that the garbage collection has ended in the selected SSD 615 of FIG. 6 through a notification from the garbage collection coordinator 210 of FIG. 2 .

20 shows embodiments of the present invention using responsive garbage collection. In embodiments of the present invention that use unresponsive garbage collection, monitor 170 of FIG. 1 monitors the scheduled start time 810 or duration 820 of FIG. 8 for garbage collection on the selected SSD of FIG. Since no selection is made, blocks 2010, 2015, and 2020 will be omitted.

FIG. 21 is similar to FIG. 20 . The difference between FIGS. 20 and 21 is that in FIG. 21 blocks 2025 and 2030 are replaced with blocks 2105 and 2110 . At block 2105 , when the scheduled start time 810 of FIG. 8 for garbage collection is received, the monitor 170 of FIG. 1 indicates that the data stored on the selected SSD 615 of FIG. 6 is now unavailable. 14 counts 1415 , 1418 , 1421 , and 1424 of each unit data 1403 , 1406 , 1409 , and 1412 of FIG. 14 in the data map 1310 of FIG. 13 of the distributed storage system to reflect can reduce The monitor 170 of FIG. 1 may know that garbage collection has started in the selected SSD 615 of FIG. 6 through a notification from the garbage collection coordinator 210 of FIG. 2 . At block 2110, when garbage collection is complete, the monitor 170 of FIG. 1 displays the data of FIG. 13 in the distributed storage system to reflect that the data stored on the selected SSD 615 of FIG. 6 is now available again. The map 1310 may increment the counts 1415 , 1418 , 1421 , and 1424 of FIG. 14 of each unit data 1403 , 1406 , 1409 , and 1412 of FIG. 14 . The monitor 170 of FIG. 1 may know that garbage collection has ended in the selected SSD 615 of FIG. 6 through a notification from the garbage collection coordinator 210 of FIG. 2 .

22A to 22B show a scheduled start time 810 and duration of garbage collection of FIG. 8 for the SSD 615 selected in FIG. 6 by the monitor of FIG. 1 , according to an embodiment of the present invention. 820) is a flowchart showing the procedure for determining In FIG. 22A , at block 2205 , the monitor 170 of FIG. 1 may receive the scheduled start time 810 of FIG. 8 as selected by the garbage collection coordinator 210 of FIG. 2 . At block 2210 , monitor 170 of FIG. 1 may receive the estimated time 725 of FIG. 7 that the selected SSD 615 of FIG. 6 needs to perform garbage collection.

Here, the flowchart may be divided according to whether the embodiments of the present invention use responsive garbage collection or non-responsive garbage collection. In embodiments of the invention that use unresponsive garbage collection, at block 2215, monitor 170 determines the scheduled start time of FIG. 8 that the selected SSD 615 of FIG. 6 needs to perform garbage collection. 810 and the estimated time 725 of FIG. 7 may be stored. Since monitor 170 does not send a response when the distributed storage system uses unresponsive garbage collection, monitor 170 of FIG. 1 here determines the scheduled start time 810 and duration 820 of FIG. to complete

In embodiments of the present invention that use responsive garbage collection, at block 2220 ( FIG. 22B ), monitor 170 may select a starting start for the selected SSD 615 of FIG. 6 to perform garbage collection. At block 2225 , the monitor 170 of FIG. 1 may select an allotted time for the selected SSD 615 to perform garbage collection. At block 2230 , the monitor 170 of FIG. 1 indicates that the selected start time, and the allotted time for the selected SSD 615 of FIG. 6 , indicates that the data stored on the selected SSD 615 of FIG. 6 will become available. At block 2235 , the monitor 170 of FIG. 1 displays the selected start time and the allocated time for the selected SSD 615 of FIG. 6 . You can check if it overlaps with other SSDs doing this garbage collection and disables many devices at the same time. If the check in one of blocks 2230 and 2235 returns a negative result (data is completely unavailable, or too many SSDs are performing garbage collection at the same time), then control returns to monitor 170 in FIG. ) returns to block 2220 to select a new start time and an assigned time. Otherwise, at block 2240, the monitor 170 may transmit the selected start time and the assigned time as the scheduled start time 810 and duration 820 of FIG. 8 to the garbage collection coordinator 210 of FIG. have.

Note that the two checks of blocks 2230 and 2235 are different. For example, it may occur that block 2230 indicates that replicated copies of data on selected SSD 615 are available in other storage devices, but because too many other SSDs are concurrently garbage collection. , block 2235 will fail. On the other hand, if only one other SSD is performing garbage collection at the same time, block 2235 may indicate that the selected SSD 615 of FIG. 6 may perform garbage collection. However, if another SSD performing the garbage collection had only another copy of some data on the selected SSD 615 of FIG. 6 , block 2230 would fail.

Also note that the arrow from block 2230 to block 2235 is marked "yes/no". If block 2230 indicates that the data will be available despite the selected SSD 615 performing garbage collection, then control will proceed to block 2235 . However, it may happen that the selected SSD 615 has a unique copy of some data on the distributed system. When this occurs, then the selected SSD 615 of FIG. 6 cannot be scheduled for garbage collection at any time without some data becoming unavailable (at least until the data is copied). In this situation, the selected SSD 615 of FIG. 6 may need to be allowed to perform garbage collection, despite the fact that some data will become unusable.

Another reason why the selected SSD 615 of FIG. 6 may be scheduled for garbage collection even though some data becomes unavailable is that the selected SSD 615 of FIG. 6 has a sufficient amount to perform garbage collection. It may be a case of waiting for the time of That is, if the selected SSD 615 of FIG. 6 exceeds some threshold time value and is waiting to perform garbage collection, even if that fact means that some data in the distributed storage system will be unavailable, The selected SSD 615 may be allowed to perform garbage collection.

22A-22B do not show the monitor 170 of FIG. 1 using the waiting list 1315 of FIG. 13 . If monitor 170 of FIG. 1 cannot schedule the selected SSD 615 of FIG. 6 for garbage collection because too many SSDs are performing garbage collection, monitor 170 of FIG. 1 indicates that fewer SSDs are garbage collected. Until collection is performed, information about the selected SSD 615 of FIG. 6 may be stored in the waiting list 1315 of FIG. 13 . Thereafter, the monitor 170 controls to remove the selected SSD 615 of FIG. 6 from the waiting list 1315 of FIG. 13 and retry scheduling garbage collection for the selected SSD 615 of FIG. 6 . may be returned to block 2220 .

If too many SSDs want to perform garbage collection at the same time, and monitor 170 of FIG. 1 may not schedule them all, then monitor 170 of FIG. algorithms can be used. Possible approaches are to randomly select an appropriate number of SSDs, to select SSDs based on the arrival times of notifications from the garbage collection coordinator 210 of FIG. 2 , to select SSDs considered to be most important (new To keep SSDs open as much as possible for data writes), or to choose SSDs that store the least amount of critical data (to keep important data available). How many SSDs can be allowed to perform garbage collection at the same time is a system parameter, and can be specified as a fixed value or a percentage value of available SSDs, statically (when a distributed storage system is deployed) or dynamic (which changes as conditions within the distributed storage system change) can be specified as

Although FIGS. 22A-22B illustrate example operation of the monitor 170 of FIG. 1 with respect to only one selected SSD 615 of FIG. 6 , the monitor 170 of FIG. 22A-22B may be performed. For example, the monitor 170 of FIG. 1 may simultaneously receive a notification that a large number of SSDs need to perform garbage collection. The monitor 170 of FIG. 1 may simultaneously perform the exemplary procedure of FIGS. 22A-22B for all SSDs.

The above description shows how I/O requests can be relocated when the storage device performs garbage collection. However, there may be situations in which it is still preferable to handle I/O requests locally even though the storage device is performing garbage collection. For example, if the storage device is only going to be garbage collected for a few more microseconds, then the time needed to communicate with another replica of the data is just enough for the storage device to complete the garbage collection and process the I/O request. It will take more time than necessary. There are also other reasons why it may be more efficient to process I/O requests on the primary replica rather than sending the I/O requests to the secondary replica. Also, there may be situations where the primary will not be garbage collected, but it may nevertheless be more efficient to handle I/O requests from the secondary rather than the primary.

23 is a view illustrating a client transmitting an I/O request to the storage node of FIG. 1 , and then relocating an I/O request to another node including a copy of the requested data according to an embodiment of the present invention; It is a block diagram showing In FIG. 23 , the client 2305 may send a read request 905 to the storage (system) node 125 . Any other I/O requests may be replaced with read requests 905 without loss of generality.

Storage node 125 (and also storage nodes 130 and 135 ) may include a storage device(s) as well as a cost analyzer 2310 and an I/O relocator 215 . In FIG. 23 , storage node 125 is capable of storing a primary copy of the data in question, so the storage device of storage node 125 is identified as primary replica 2315 . On the other hand, since storage nodes 130 and 135 store backup copies of the data in question, the storage devices of storage nodes 130 and 135 are identified as secondary replicas 2320 and 2325 .

As described below with reference to FIGS. 24-38 , the cost analyzer 2310 calculates costs associated with processing I/O requests locally (at the primary replica) and remotely (at one of the secondary replicas). can decide In this context, “cost” can be interpreted as “time”. That is, it will take some time to process the I/O request 905 from the primary replica 2315 associated with one of the secondary replicas 2320 and 2325. However, in other embodiments of the present invention, “cost” may refer to a concept other than time, such as I/O performance metrics (IOPS, latency, throughput, etc.), different service levels, etc. . The I/O relocator 215 may then compare the cost of performing the I/O request 905 on the primary replica 2315 with the secondary replicas 2320 and 2325 and select the replica accordingly. .

24 is a block diagram specifically illustrating the cost analyzer 2310 of FIG. 23 . In FIG. 24 , the cost analyzer 2310 includes a local (local) time estimator 2405 , a remote time estimator 2410 , query logic 2415 , reception logic 2420 , database 2425 , local forecast analyzer 2435 . , and a remote prediction analyzer 2435 . The local time estimator 2405 and the remote time estimator 2410 perform the I/O request 905 of FIG. 23 at the primary replica 2315 of FIG. 23 and the secondary replicas 2320 and 2325 of FIG. 23 based on the current information. ), you may want to calculate the time required to process it. A local time estimator 2405 and a remote time estimator 2410 are further described below with reference to FIGS. 26 and 27, respectively, below.

Query logic 2415 and receive logic 2420 may be used to send requests for information and receive responses to the requests. For example, as described below with reference to FIG. 28 , the query logic 2415 may query the primary replica 2315 of FIG. 23 for the number of empty pages, and the receive logic 2420 of FIG. It may receive the number of blank pages from the primary replica 2315 . Query logic 2415 calculates the threshold number of empty pages and number of empty pages of primary replica 2315 or secondary replicas 2320 and 2325 of FIG. 23 , the pending I/ the number of O requests and the time required to process each I/O request, the time required to communicate with the (storage) system nodes 130 and 135, including the secondary replicas 2320 and 2325 of FIG. 23, and Information such as remote processor load and remote software stack load from processors associated with system nodes 130 and 135, including auxiliary replicas 2320 and 2325 of FIG. 23, to the replicas for end use. As needed to keep current information (i.e., the local time estimator 2405 needs to calculate the local estimated time required to process the I/O request 905 of FIG. 23 on the primary replica 2315 of FIG. ) or periodically (generally every 10 seconds, every few seconds, every second, or at some regular interval, such as every fraction of a second).

Database 2425 contains information used by the modules of various cost analyzer 2310 , such as local time estimator 2405 , remote time estimator 2410 , local forecast analyzer 2430 , and remote forecast analyzer 2435 . can be saved. Database 2425 is further described with reference to FIG. 31 below. Finally, the local prediction analyzer 2430 and the remote prediction analyzer 2435 can make predictions regarding the time required to process the I/O request 905 of FIG. 23 either locally or remotely based on historical data. have. Local prediction analyzer 2430 and remote prediction analyzer 2435 are further described with reference to FIGS. 32-36, respectively, below.

FIG. 25 is a block diagram specifically illustrating the I/O relocation 215 of FIG. 23 . In FIG. 25 , I/O relocator 215 may include storage 2505 , a first comparator 2510 , a second comparator 2515 , and a selector 2520 . Storage 2505 may store information such as threshold time 2525 . Threshold time 2525 may be a threshold time that has no value when sending I/O request 905 of FIG. 23 to the secondary replica thereafter. For example, threshold time 2525 may be an amount of time less than the time required to communicate with secondary replicas 2320 and 2325 of FIG. 23 . 23 at secondary replicas 2320 and 2325 of FIG. 23 when the time required for the primary replica 2315 of FIG. 23 to process the I/O request 905 of FIG. 23 is less than the threshold time 2525 It is not even necessary to calculate the time required to process the /O request 905 . The I/O relocator 215 uses the first comparator 2510 to make this determination to estimate the estimated time required to process the I/O request 905 of FIG. 23 on the primary replica 2315 of FIG. can be compared.

The first comparator 2510 calculates the estimated time required to process the I/O request 905 of FIG. 23 at the primary replica 2315 of FIG. 2 at the secondary replicas 2320 and 2325 of FIG. This can be compared to the estimated time required to process the O request 905 . The selector 2520 may then select one of the primary replica 2315 of FIG. 23 and the secondary replicas 2320 and 2325 of FIG. 23 based on the results of the second comparator 2515 .

FIG. 26 is a block diagram specifically illustrating the local time estimator 2405 of FIG. 24 . The local time estimator 2405 may estimate how long it will take for the primary replica 2315 of FIG. 23 to satisfy the I/O request 905 of FIG. 23 . To support this operation, the local time estimator 2405 includes a local garbage collection time calculator 2605 , a local estimated garbage collection time calculator 2610 , a queue processing time calculator 2615 , storage 2620 , and a local estimated time spent a calculator 2625 , and a weight generator 2630 . The local garbage collection time calculator 2605 may calculate a local garbage collection time for the primary replica 2315 of FIG. 23 when the primary replica 2315 of FIG. 23 is currently performing garbage collection. Local Estimated Garbage Collection Time Calculator 2610 calculates Local Garbage Collection Time, except that Local Estimated Garbage Collection Time Calculator 2610 estimates how long it will take for the primary replica 2315 of FIG. 23 to perform an upcoming garbage collection. Similar to calculator 2605 . Queue processing time calculator 2615 may calculate the time it will take for primary replica 2315 of FIG. 23 to process I/O requests 905 in the queue (primary replica 2315 of FIG. Assuming processing in the order received, this will be completed before I/O request 905 in FIG. 23 is processed). Storage 2620 may store information, such as local garbage collection weights 2635 , local expected garbage collection weights 2640 , and queue processing weights 2645 , used by local time estimator 2405 . These weights, which may be generated by weight generator 2630 , are further described below with reference to FIG. 33 . Finally, the local estimated time required calculator 2625 may generate an estimate of the time required to complete the processing of the I/O request 905 of FIG. 23 based on prior information rather than the actual current information.

FIG. 27 is a block diagram specifically illustrating the remote time estimator 2410 of FIG. 24 . The remote time estimator 2410 may estimate the time it will take the secondary replicas 2320 and 2325 of FIG. 23 to satisfy the I/O request 905 of FIG. 23 . To support this operation, remote time estimator 2410 includes communication time calculator 2705 , remote processing time calculator 2710 , remote garbage collection time calculator 2715 , storage 2720 , and remote estimated time spent calculator 2725 . ), and a weight generator 2730 . The communication time calculator 2705 may calculate the time required for communication with the secondary replicas 2320 and 2325 of FIG. 23 . The remote processor time calculator 2710 calculates the I/ of FIG. 23 , based on current loads on the processors of system nodes 130 and 135 (including auxiliary replicas 2320 and 2325 , respectively) in FIG. 23 . We can calculate the time required to process the O request 905 . Assuming that secondary replicas 2320 and/or 2325 of FIG. 23 are currently performing garbage collection, remote garbage collection time calculator 2715 calculates the remote garbage collection time for secondary replicas 2320 and 2325 of FIG. 23 . can be calculated. Storage 2720 may store information used by remote time estimator 2405 , such as communication time weights 2735 , remote processor time weights 2740 , and remote garbage collection weights 2745 . These weights, which may be generated by weight generator 2630 , are further described below with reference to FIG. 37 . Finally, remote estimated time required calculator 2725 may generate an estimate of the time required to complete processing of I/O request 905 of FIG. 23 based on prior information rather than actual current information.

28 and 29 are block diagrams illustrating the local garbage collection time calculator 2605 and the local expected garbage collection calculator 2610 of FIG. 26 for calculating a local garbage collection time and a local expected garbage collection time. Since the operations of these two calculators are similar to each other, they will be described together. The only difference between the operations of the two calculators is that the local garbage collection time calculator 2605 determines the time required to complete a garbage collection operation already in progress on the primary replica 2315 of FIG. 23 , whereas the local estimated garbage collection time calculator 2605 . 2610 determines the time required to perform a garbage collection operation that is scheduled to start (but has not actually started yet).

In each case, the query logic 2415 of FIG. 24 may query the primary replica 2315 of FIG. 23 for the actual number of empty pages 2805 and the empty page threshold 2810 . (Instead, the empty page threshold 2810 is generally a constant value for a given model of storage, so rather than querying the primary replica 2315 of FIG. It can be stored and accessed from 2620 ). The number of empty pages 2805 may indicate how many empty pages are currently present in the primary replica 2315 of FIG. 23 . The blank page threshold 2810 may indicate the minimum number of blank pages required in the primary replica 2315 of FIG. 23 . When the number of empty pages 2805 drops below the empty page threshold 2810 , the primary replica 2315 of FIG. 23 will perform a garbage collection to free more pages for data writes.

Once the receiving logic 2420 of FIG. 24 has received the number of empty pages 2805 and the empty page threshold 2810, the local garbage collection time calculator 2605 and the local expected garbage collection time calculator 2610 An average garbage collection time 2815 may be determined. The local average garbage collection time 2815 may indicate an average amount of time required to empty one page in the primary replica 2315 of FIG. 23 . The local average garbage collection time 2815 can be determined by accessing a fixed value that can be stored in the storage 2620 of FIG. 26 or the database 2425 of FIG. 24 , and also at the primary replica 2315 of FIG. It can be calculated from previous information about garbage collection operations. Previous information may be stored in the database 2425 of FIG. 24 .

Although the local average garbage collection time 2815 includes the term “average” in its name, the local average garbage collection time 2815 may be calculated in any desired manner. For example, the local average garbage collection time 2815 is the average, median, or mode of the time it takes to recover one page for all garbage collection operations performed on the primary replica 2315 of FIG. 23 . ) can be calculated as Alternatively, the local average garbage collection time 2815 may be calculated using linear regression analysis on previous local garbage collection information of the primary replica 2315 of FIG. 23 . Alternatively, the local average garbage collection time 2815 is based on a sliding window of the most recent garbage collection operations (eg, the most recent 10 (or any other desired number) garbage collection operations). can be calculated. Other techniques for calculating the local average garbage collection time 2815 may still be used.

The local garbage collection time calculator 2605 and the local expected garbage collection time calculator 2610 calculate the actual number of empty pages 2805 on the primary replica 2315 of FIG. 23 multiplied by the local average garbage collection time 2815 and A local garbage collection time 2820 and a local expected garbage collection time 2905 may be calculated as the difference between the empty page threshold value 2810 .

Optionally, local garbage collection time calculator 2605 and local expected garbage collection time calculator 2610 can account for the time required to program valid pages of erase blocks into other pages before they are erased. A delay 2825 may also be included. The programming delay 2825 , like the local average garbage collection time 2815 , may be a predetermined fixed value or may be calculated from previous information in much the same way as the local average garbage collection time 2815 . The programming delay 2825 may be included as a constant value in the local garbage collection time 2820 and the local expected garbage collection time 2905, or the difference between the actual number of empty pages 2805 and the empty page threshold 2810 may be multiplied (to account for the fact that the number of pages requiring programming may vary).

Local garbage collection time calculator 2605 and local expected garbage collection time calculator 2610 operate very similarly, in embodiments of the invention, which may be implemented using one logic that includes both variants. Local garbage collection time calculator 2605 and local expected garbage collection time calculator 2610 are implemented using logic circuits or software running on a processor (eg, an In-Storage Processor on an SSD, respectively), respectively. can be Also, remote garbage collection time calculator 2715 of FIG. 27 is similar to local garbage collection time calculator 2805, except that queries relate to the garbage collection status of secondary replicas 2320 and 2325 of FIG. It works. Therefore, the remote garbage collection time calculator 2715 of FIG. 27 can be understood based on the description of the local garbage collection time calculator 2605 , and further detailed descriptions are omitted.

30 is a block diagram illustrating the queue processing time calculator 2615 of FIG. 26 for calculating a queue processing time. The queue processing time calculator 2615 obtains the number of pending I/O requests 3005 and the time taken to process one I/O request 3010 to determine the queue processing time 3015, values can be multiplied.

Query logic 2415 of FIG. 24 may query primary replica 2315 of FIG. 23 for the number of pending requests that receive logic 2420 of FIG. 24 may receive. The required time 3010 is generally a fixed value and may be stored in the database 2425 of FIG. 24 or the storage 2620 of FIG. 26 , but the time required 3010 is determined from previous information stored in the database 2425 of FIG. 24 . may be calculated. Much like the local average garbage collection time 2815 of FIG. 28 , the time taken 3010 is the average, median value of the times required to previously perform an I/O instruction on the primary replica 2315 of FIG. 23 , Or it can be calculated as the mode, or it can be calculated from this previous information using linear regression analysis. Also, the previous information used may include a sliding window of all previous data or just previous information.

FIG. 31 is a block diagram specifically illustrating the database 2425 of FIG. 24 . In Fig. 31, database 2425 is shown to store various information. This information may include the following information.

Historical Local Garbage Collection Information 3105: When garbage collection is performed on the primary replica 2315 of FIG. 23, historical information on how long the garbage collection took.

Worst case local garbage collection information 3110: When garbage collection is performed on the primary replica 2315 of Article 23, the time taken for garbage collection in the worst case.

Average case local garbage collection information 3115: When garbage collection is performed on the primary replica 2315 of FIG. 23, the average time it takes for garbage collection.

Previous Processing Time Information 3120: Previous information regarding the time required for the primary replica 2315 of FIG. 23 to process one I/O request 905

Worst case processing time information 3125: In the worst case, the time required for the primary replica 2315 of FIG. 23 to process one I/O request 905

Average case processing time information 3130: on average, the time required for the primary replica 2315 of FIG. 23 to process one I/O request 905

Previous Communication Time Information 3135: Previous information regarding the time taken to communicate with the secondary replicas 2320 and 2325 of FIG. 23 .

Worst case communication time information 3140: The time taken to communicate with the secondary replicas 2320 and 2325 of FIG. 23 in the worst case.

Average Case Communication Time Information 3145: The average time taken to communicate with the secondary replicas 2320 and 2325 of FIG. 23 .

Previous remote processor time information 3150: Previous information regarding the processor and software stack loads (loads) on the processors associated with system nodes 130 and 135 including secondary replicas 2320 and 2325 of FIG. 23 . , and how it affects the time required to process one I/O request 905 in secondary replicas 2320 and 2325 of FIG. 23 .

Worst case remote processor time information 3155: In the worst case, the time of remote processor and software stack loads of processors associated with system nodes 130 and 135 including secondary replicas 2320 and 2325 of FIG. 23 . effect.

Remote Processor Time Information 3160 in Average Case: The average temporal impact of remote processor and software stack loads of processors associated with system nodes 130 and 135 including auxiliary replicas 2320 and 2325 of FIG. 23 .

Previous remote garbage collection information 3165: When garbage collection is performed on the secondary replicas 2320 and 2325 of FIG. 23, previous information about the time it took for the garbage collection to take place.

Worst case remote garbage collection information 3170: When garbage collection is performed on secondary replicas 2320 and 2325 of FIG. 23, the time taken for garbage collection in the worst case.

Average case remote garbage collection information 3175: When garbage collection is performed on the secondary replicas 2320 and 2325 of FIG. 23, the average time taken for garbage collection.

As can be seen from a quick review of information that may be stored in database 2425 , some of this information relates to local time estimator 2405 of FIG. 24 , and some of this information relates to remote time estimator 2410 of FIG. 24 . is related to 24 and 31 suggest that all such information is stored in one database (ie, database 2425), embodiments of the present invention divide this information into multiple databases and store the information in various locations. can For example, information 3105 - 3130 may be stored in a database within local time estimator 2405 of FIG. 24 (which may be within storage 2620 of FIG. 26 ), and information 3135 - 3175 may be It may be stored in a database within the remote time estimator 2410 of FIG. 24 (which may be within storage 2720 of FIG. 27 ).

32 is a block diagram specifically illustrating the local prediction analyzer 2430 of FIG. 24 . The local garbage collection time calculator 2605, the local expected garbage collection time calculator 2610, and the queue processing time calculator 2615 of FIG. 26 all indicate that the primary replica 2315 of FIG. The current information about the primary replica 2315 of FIG. 23 can be used to estimate the time it will take to process 905 . On the other hand, the local prediction analyzer 2430 can estimate the time required for the primary replica 2315 of FIG. 23 to process the I/O request 905 of FIG. 23 based only on previous information. Using previous information in this manner counterpoints the information provided by local garbage collection time calculator 2605, local expected garbage collection time calculator 2610, and queue processing time calculator 2615 of FIG. can provide

In addition, the local prediction analyzer 2430 provides that the primary replica 2315 is required by the local garbage collection time calculator 2605, the local expected garbage collection time calculator 2610, and the queue processing time calculator 2615 of FIG. In situations where it is not possible to provide the required information, the primary replica 2315 of FIG. 23 may provide the estimated time needed to process the I/O request 905 of FIG. 23 . For example, if the primary replica 2315 does not provide information about the number of empty pages 2805 of FIG. 28 , then the local garbage collection time calculator 2605 and the local expected garbage collection time calculator 2610 of FIG. may not estimate the time required to perform garbage collection on the primary replica 2315 of FIG. 23 . In contrast, the local prediction analyzer 2430 uses only previous information and does not rely on having access to information from the primary replica 2315 of FIG. 23 .

The local prediction analyzer 2430 can access information from the database 2425 and use that information to determine the time it takes for the primary replica 2315 of FIG. 23 to complete the I/O request 905 of FIG. It can be used to generate a predictable, expected local time 3205 . For example, the local prediction analyzer 2430 can obtain historical information regarding the time it took the primary replica 2315 of FIG. 23 to process in the past, and the primary replica 2315 of FIG. The information may be used to make an estimate regarding the time it will take to process the request 905 . Note that because the local prediction analyzer 2430 uses older information rather than the current information about the primary replica 2315 of FIG. 23 , the estimated local time 3205 may not be accurate. If the primary replica 2315 of FIG. 23 is busier than in the past (eg, if the primary replica 2315 of FIG. 23 needs to perform a greater than normal amount of garbage collection), the estimated local time 3205 is It may be less than the actual time required. On the other hand, if the primary replica 2315 of FIG. 23 is not as busy as in the past (eg, the primary replica 2315 of FIG. 23 is busy processing a small amount of pending I/O requests, but does not need to perform garbage collection). If not), the estimated local time 3205 may be greater than the actual required time.

The local prediction analyzer 2430 may calculate the expected local time 3205 from the information in the database 2425 in any desired manner. For example, the local prediction analyzer 2430 may calculate the average value, median value, or mode of the previous local garbage collection information 3105 of FIG. 31 , and the average value of the previous processing time information 3120 of FIG. 31 , The median or mode can be computed, and then the two statistical calculations can be combined using weighted summation. Alternatively, the local prediction analyzer 2430 may consider only the previous local garbage collection information 3105 of FIG. 31 and ignore any processing time information. Alternatively, the local prediction analyzer 2430 may consider only the sliding window of information in the database 2425 . Embodiments of the present invention are intended to include all variations on how the local prediction analyzer 2430 computes the expected local time 3205 .

33 is a block diagram specifically illustrating the local estimated required time calculator 2625 of FIG. 26 . 33, the local estimated time spent calculator 2625 can obtain information such as a local garbage collection time 2820, a local estimated garbage collection time 2905, and a queue processing time 3015, and a local estimated time taken ( 3305), we can combine them. The local estimated time required calculator 2625 may include a local garbage collection weight 2635 , a local expected garbage collection weight 2640 , and a queue processing weight 2645 . These weights may indicate how important each corresponding time is considered in the calculation of the local estimated time taken 3305 . For example, each of the times may be multiplied by a corresponding weight, and the results may be added together to calculate the local estimated time taken 3305 .

The local garbage collection weight 2635 , the local expected garbage collection weight 2640 , and the queue processing weight 2645 may be calculated by any desired means. For example, linear regression analysis may be performed on information on the database 2425 of FIG. 24 to calculate weights of the weight generator 2630 of FIG. 26 . This linear regression analysis may be performed on all information in the database 2425 of FIG. 24 , or may be performed on a sliding window of information in the database 2425 .

The local estimated time spent calculator 2625 calculates the estimated local time 3205 , which may also be optionally weighted by the local expected weights 3310 (which may also be generated by the weight generator 2630 of FIG. 26 ). can also be considered. By taking the estimated local time 3205 into account, the local estimated time-spent calculator 2625 can counteract anomalous information coming from the primary replica 2315 of FIG. 23 that may lead to unusually low or high estimated times.

Although the above description uses weights 2635, 2640, 2645, and 3310, the weighted calculation is optional. For example, the local estimated time required calculator 2625 may calculate the sum of values without applying any weights. In other words, weights 2635 , 2640 , 2645 , and 3310 may all be implied weights rather than actually stored in storage 2620 of FIG. 26 . In a similar manner, if the local estimated time spent calculator 2625 is designed to calculate the local estimated time spent 3305 using only a subset of available values (eg, only the local garbage collection time 2820 ). , "weights" applied to other values may be set to zero, to avoid such values affecting the result. Again, in this situation, the weights may imply the following descriptions. Local garbage collection weight 2635 may be implicitly equal to one, and weights 2640 , 2645 , and 3310 may be zero. Of course, in situations where the weight is set to zero, the corresponding value does not need to be calculated at first, and the corresponding components generating the value may be appropriately omitted from the embodiments of the inventive concept.

FIG. 34 is a block diagram specifically showing the communication time calculator 2705 of FIG. 27 . Communication time calculator 2705 may check the communication status of system nodes 130 and 135 including secondary replicas 2320 and 2325 of FIG. 23 to determine the time required to communicate with the nodes. (ping) logic 3405 . Alternatively, communication time calculator 2705 can access previous communication time information 3135 from database 2425 of FIG. 31 , worst case communication time 3140 , and average case communication time 3145 and , that information can be used to calculate the communication time 3410 . Embodiments of the present invention may also include other approaches for calculating communication time with system nodes 130 and 135 . As an example, the ping logic 3405 may send a small data request to the secondary replicas 2320 and 2325 of FIG. 23 and use the small data request to communicate with the secondary replicas 2320 and 2325 of FIG. time can be measured.

FIG. 35 is a block diagram specifically illustrating the remote processor time calculator 2710 of FIG. 27 . In FIG. 35 , remote processor time calculator 2710 can obtain remote processor load 3505 and remote software stack load 3510 . Query logic 2415 and receive logic 2420 of FIG. 24 load 3505 remote processor from processor associated with system nodes 130 and 135 of FIG. 23 including secondary replicas 2320 and 2325 of FIG. and remote software stack loads 3510 . Remote processor load 3505 may represent a load on the processor of system nodes 130 and 135 of FIG. 23 , while remote software stack load 3510 may represent a load on the processor of system nodes 130 and 135 of FIG. 23 . It may represent loads on the software being executed. The remote processor time calculator 2710 may use any desirable approach to convert loads 3505 and 3510 to remote processor time 3515 . Further, remote processor time calculator 2710 may calculate remote processor time 3515 based on only one of two loads 3505 and 3510 rather than both loads 3505 and 3510 .

FIG. 36 is a block diagram specifically illustrating the remote prediction analyzer 2435 of FIG. 24 . Communication time calculator 2705, remote processor time calculator 2710, and remote garbage collection time calculator 2715 of FIG. 27 allow secondary replicas 2320 and 2325 of FIG. We can use the current information about the secondary replicas 2320 and 2325 of FIG. 23 to estimate the time it takes to process . On the other hand, the remote prediction analyzer 2435 may estimate the time required for the secondary replicas 2320 and 2325 of FIG. 23 to process the I/O request 905 of FIG. 23 based only on the previous information. Using previous information in this manner may provide a counterpoint to the information provided by communication time calculator 2705 , remote processor time calculator 2710 , and remote garbage collection time calculator 2715 of FIG. 27 .

In addition, the remote prediction analyzer 2435 provides that auxiliary replicas 2320 and 2325 are required by the communication time calculator 2705, the remote processor time calculator 2710, and the remote garbage collection time calculator 2715 of FIG. In situations that do not provide the required information, the secondary replicas 2320 and 2325 of FIG. 23 may provide the estimated time needed to process the I/O request 905 of FIG. 23 . For example, if secondary replicas 2320 and 2325 do not provide information about the number of empty pages 2805 of FIG. 28 , remote garbage collection time calculator 2715 of FIG. At (2320 and 2325), the time required to perform garbage collection may not be estimated. In contrast, the teleprediction analyzer 2435 uses only previous information and does not rely on being able to access information from the secondary replicas 2320 and 2325 of FIG. 23 .

The teleprediction analyzer 2435 can access information from the database 2425 and estimate the time it will take for secondary replicas 2320 and 2325 of FIG. 23 to complete the I/O request 905 of FIG. , may use that information to generate an estimated remote time 3605 . For example, the teleprediction analyzer 2435 can obtain historical information regarding the time it took the auxiliary replicas 2320 and 2325 of FIG. 23 to process in the past, and the auxiliary replicas 2320 and 2325 of FIG. 23 . That information can be used to make an estimate regarding the time it will take to process the I/O request 905 of this figure. Note that the estimated remote time 3605 may not be accurate because the remote prediction analyzer 2435 uses older information rather than the current information about the auxiliary replicas 2320 and 2325 of FIG. 23 . If secondary replicas 2320 and 2325 of FIG. 23 are busier than in the past (eg, secondary replicas 2320 and 2325 of FIG. 23 need to perform a larger amount than their normal garbage collection amount), the expected The remote time 3605 may be shorter than the actual required time. On the other hand, if secondary replicas 2320 and 2325 of FIG. 23 are not as busy as in the past (eg, secondary replicas 2320 and 2325 of FIG. 23 are busy processing a small number of pending I/O requests, but garbage If there is no need to perform the collection), the estimated remote time 3605 may be longer than the actual required time.

The remote prediction analyzer 2435 may calculate the expected remote time 3605 from this information in the database 2425 in any desired manner. For example, the remote prediction analyzer 2435 may calculate an average value, a median value, or a mode of the previous remote garbage collection information 3165 of FIG. 31 , the average value of the previous communication time information 3135 of FIG. 31 , One can compute the median, or mode, and combine the two statistical calculations using weighted summation. Alternatively, the remote prediction analyzer 2435 may consider only the previous remote garbage collection information 3165 of FIG. 31 and ignore the communication time and remote processor time information. Alternatively, the remote prediction analyzer 2435 may consider only the sliding window of information in the database 2425 . Embodiments of the present invention may be intended to include all variations of the method by which the remote prediction analyzer 2435 calculates the expected remote time 3605 .

FIG. 37 is a block diagram specifically illustrating the remote estimation required time calculator 2725 of FIG. 27 . In FIG. 37 , a remote estimated time spent calculator 2725 can obtain information such as a communication time 3410 , a remote processor time 3515 , and a remote garbage collection time 3705 , and calculates a remote estimated time spent 3710 . You can combine them to calculate. The remote estimated time required calculator 2725 may also include a communication time weight 2735 , a remote processor time weight 2740 , and a remote garbage collection time weight 2745 . These weights may indicate how important each corresponding time is considered in the calculation of the remote estimated time taken 3710 . For example, each of the times may be multiplied by a corresponding weight, and the results may be summed together to calculate the remote estimated time required 3710 .

Communication time weight 2735 , remote processor time weight 2740 , and remote garbage collection time weight 2745 may be calculated by any desired means. For example, the weight generator 2730 of FIG. 27 may perform linear regression analysis on information on the database 2425 of FIG. 24 to calculate weights. This linear regression analysis may be performed on all information in database 2425 of FIG. 24 , or it may be performed on a sliding window of information in database 2425 .

The remote estimated time required calculator 2725 may take into account the estimated remote time 3605 , which may optionally be weighted by the remote estimated weights 3715 (which may also be generated by the weight generator 2730 of FIG. 27 ). . By taking into account the estimated remote time 3605, the estimated remote time 2725 cancels out anomalous information from the secondary replicas 2320 and 2325 of FIG. can do it

Although the above description uses weights 2735, 2740, 2745, and 3715, the weighted calculation is optional. For example, the remote estimated time required calculator 2725 may calculate the sum without applying any weights to the values. In other words, weights 2735 , 2740 , 2745 , and 3715 may be implied weights rather than actually stored in storage 2720 of FIG. 27 . In a similar manner, if the remote estimated time spent calculator 2725 is designed to calculate the remote estimated time spent 3710 using only a subset of the available values (eg, only the remote garbage collection time 3705), other values The “weights” applied to can be set to zero to avoid affecting the result. Again, in this situation, weights can be implied. The remote garbage collection weight 2745 may be implicitly one, and the weights 2735 , 2740 and 3715 may be zero. Of course, in these situations where the weight is set to zero, the corresponding value does not need to be calculated at first, and components generating the corresponding value may be appropriately omitted from the embodiments of the present invention.

38 is a block diagram specifically illustrating the I/O relocation 215 of FIG. 25 . In FIG. 38 , I/O relocation 215 may receive a threshold time 2525 and a local estimated time taken 3305 . The first comparator 2510 may calculate these values. As indicated in box 3805, if the local estimated time taken 3305 is less than the threshold time 2525, then the I/O relocator 215 redirects the I/O request 905 of FIG. can be sent to the replica 2315. Otherwise, the local estimated time taken 3305 may be passed to a second comparator 2515 , which may also receive a remote estimated time taken 3710 for each secondary replica 130 and 135 of FIG. 23 . have. The selector 2520 then selects the selector 2520, based on whether the I/O relocator 215 has the least estimated time required to later send the I/O request 905 of FIG. 23 to the selected replica. You can select one of secondary replicas 2320 and 2325 and primary replica 2315 of (one of 2320 and 2325).

Figures 39A-39B show where the cost analyzer 2310 and the I/O relocator 215 of Figure 23 determine where to send the I/O request 905 of Figure 23, in accordance with an embodiment of the present invention. It is a flow chart showing the procedure for At block 3905 of FIG. 39A , the system note 125 of FIG. 23 may receive the I/O request 905 of FIG. 23 . At block 3910 , I/O relocation 215 of FIG. 23 may determine whether primary replica 2315 of FIG. 23 is currently performing garbage collection. If the primary replica 2315 of FIG. 23 is not currently performing garbage collection, then at block 3915 , the I/O relocation 215 of FIG. 23 redirects the I/O request 905 of FIG. 23 to the primary replica 2315 of

If the primary replica 2315 of FIG. 23 is currently performing garbage collection, at block 3920 , the local estimated time spent calculator 2625 of FIG. 26 may calculate the local estimated time spent 3305 of FIG. 33 . . In block 3925 , the first comparator 2510 may compare the local estimated required time 3305 of FIG. 33 with the threshold time 2525 of FIG. 25 . In block 3930 , the first comparator 2510 may determine whether the local estimated required time 3305 of FIG. 33 is shorter than the threshold time 2525 of FIG. 25 . If the local estimated time required 3305 of FIG. 33 is less than the threshold time 2525 of FIG. 25, then at block 3915, the process returns to the I/O relocation 205 of FIG. 23 with the I/O request ( 905) to the primary replica 2315 of FIG. 23 may continue.

On the other hand, if the local estimated required time 3305 of FIG. 33 is longer than the threshold time 2525 of FIG. 25 , then at block 3935 ( FIG. 39B ), the cost estimator 2310 of FIG. Select one of (2320 and 2325). At block 3940 , the remote estimated time required calculator 2725 of FIG. 27 may calculate the remote estimated time required 3710 of FIG. 37 . At block 3945 , the cost estimator 2310 of FIG. 23 may determine whether there are more secondary copies of the data requested by the I/O request 905 of FIG. 23 . If there are more secondary replicas, processing may return to block 3935 to compute the remote estimated time taken 3710 of FIG. 37 for the other secondary replicas. Otherwise, at block 3950, the second comparator 2515 compares the local estimated duration 3305 of FIG. 3710) can be compared. At block 3955 , selector 2520 may select one of primary replica 2315 and secondary replicas 2320 and 2325 of FIG. 23 based on which one has the lowest estimated turnaround time associated. Finally, at block 3960 , the I/O relocator 215 of FIG. 23 may relocate the I/O request 905 of FIG. 23 to the selected replica, after which processing is complete.

39A to 39B , the flowchart is for sending the I/O request 905 of FIG. 23 to the primary replica 2315 of FIG. 23 when the primary replica 2315 of FIG. 23 is not currently performing garbage collection. 23 shows the I/O relocation 215 . Garbage collection is often the main reason why it is more efficient to send the I/O request 905 of FIG. 23 to one of the secondary replicas 2320 and 2325 of FIG. Therefore, when the primary replica 2315 of FIG. 23 is not performing garbage collection, the I/O request 905 of FIG. 23 can often be most efficiently handled by the primary replica 2315 of FIG. However, in some embodiments of the present invention, block 3910 is omitted, and even if primary replica 2315 of FIG. 23 performs garbage collection, cost estimator 2310 of FIG. 23 calculates local and remote time spent. can be calculated

40 is a flowchart illustrating a procedure for calculating the local estimated required time 3305 of FIG. 33 by the local estimated required time calculator 2625 of FIG. 26 according to an embodiment of the present invention. At block 4005 of FIG. 40 , the local garbage collection time calculator 2605 of FIG. 26 may calculate the local garbage collection time 2820 of FIG. 28 . At block 4010 , the local expected garbage collection time calculator 2610 of FIG. 26 may calculate the local expected garbage collection time 2905 of FIG. 29 . At block 4015 , the queue processing time calculator 2615 of FIG. 26 may calculate the queue processing time 3105 of FIG. 30 . At block 4020 , the local prediction analyzer 2430 of FIG. 24 may calculate the expected local time 3205 of FIG. 32 . At block 4025 , the weight generator 2630 of FIG. 26 may calculate weights applied to various times, which are used in calculating the local estimated time taken 3305 of FIG. 33 . Finally, at block 4030 , the local estimated time required calculator 2625 of FIG. 26 may calculate the local estimated time required 3305 of FIG. 33 from the various times and weights.

41A-41B illustrate a local garbage collection time calculator 2605 and a local expected garbage collection time calculator 2610 of FIG. 26, and a remote garbage collection time calculator 2715 of FIG. 27, according to an embodiment of the present invention. is a flowchart showing a procedure for calculating the local garbage collection time 2820 of FIG. 28 , the local expected garbage collection time 2905 of FIG. 29 , and the remote garbage collection time 3705 of FIG. 37 . For simplicity of explanation, with reference to FIGS. 41A-41B , all references to the local garbage collection time calculator 2605 of FIG. 26 are referenced to the local estimated garbage collection time calculator 2610 of FIG. 26 and the remote garbage collection of FIG. 27 . Reference is also made to the time calculator 2715 . All references to primary replica 2315 of FIG. 23 are intended to also refer to secondary replicas 2320 and 2325 of FIG. 23 . All references to local average garbage collection time 2815 in FIG. 28 are intended to also refer to remote average garbage collection time. Further, all references to local garbage collection time 2805 of FIG. 28 are intended to also refer to local expected garbage collection time 2905 of FIG. 29 and remote garbage collection time 3705 of FIG. 37 .

At block 4105 of FIG. 41A , the local garbage collection time calculator 2605 of FIG. 26 may check whether the primary replica 2315 of FIG. 23 is or will be performing garbage collection. This check may be completed by comparing the number of empty pages 2805 of FIG. 28 to the empty page threshold 2810 of FIG. If the number of empty pages 2805 of FIG. 28 is less than the empty page threshold 2810 of FIG. 28 , the primary replica 2315 of FIG. 23 is performing or is about to start garbage collection.

If the primary replica 2315 of FIG. 23 is not performing garbage collection or is not starting soon, the local garbage collection time calculator 2605 of FIG. 26 returns the local garbage collection time 2820 of FIG. 28 as zero. can do. Otherwise, the primary replica 2315 of FIG. 23 is performing garbage collection or is starting soon.

At this point, there are two possible approaches you can take. One approach is to use previous information about garbage collection in the primary replica 2315 of FIG. 23 , stored in the database 2425 of FIG. 24 . At block 4110 , the local garbage collection time calculator 2605 of FIG. 26 can access previous information in the database 2425 of FIG. 24 , and at block 4115 , the local garbage collection time calculator 2605 of FIG. 26 . may calculate the local garbage collection time 2820 of FIG. 28 using the previous information.

Another approach is shown in FIG. 41B . At block 4120 , the query logic 2415 of FIG. 24 may query the number of empty pages 2805 of FIG. 28 . As indicated by dashed arrow 4125 , block 4120 may be repeated as needed. This is because the query logic 2415 of FIG. 24 is set up to make a query on a regular basis, or there are multiple replicas to query (eg, to calculate all possible remote garbage collection times 2905 of FIG. 29 ). There may be multiple secondary replicas to do). At block 4130 , the receive logic 2420 of FIG. 24 may receive the number of empty pages 2805 of FIG. 28 from the replica(s). In block 4135 , the query logic 2415 of FIG. 24 may query the empty page threshold 2810 of FIG. 28 . As indicated by dashed arrow 4140 , block 4135 may be repeated as needed. This is because the query logic 2415 of FIG. 24 is set up to query regularly, or there are multiple replicas to query (eg, query to calculate all possible remote garbage collection times 2905 of FIG. 29 ). There may be multiple secondary replicas). At block 4145 , the receive logic 2420 of FIG. 24 may receive the empty page threshold(s) 2810 of FIG. 28 from the replica(s).

At block 4150 , the local garbage collection time calculator 2625 of FIG. 26 may calculate a difference value between the number of empty pages 2805 of FIG. 28 and the empty page threshold 2810 of FIG. 28 . This difference value may indicate the number of pages that must be emptied to bring the primary replica 2315 of FIG. 23 out of the garbage collection state. At block 4155 , the local garbage collection time calculator 2625 of FIG. 26 may add a delay associated with programmable valid pages of blocks being erased. At block 4160, the local garbage collection time calculator 2625 of FIG. 26 may calculate the local garbage collection time 2820 of FIG. 28 by multiplying the above result by the local average garbage collection time 2815 of FIG.

42 is a flowchart illustrating a procedure for calculating the queue processing time 3015 of FIG. 30 by the queue processing time calculator 2615 of FIG. 26 according to an embodiment of the present invention. At block 4205 of FIG. 42 , the query logic 2415 of FIG. 24 may request the queue depth (ie, the number of pending I/O requests 3005 of FIG. 30 ) from the primary replica 2315 of FIG. have. At block 4210 , the receive logic 2420 of FIG. 24 may receive the queue depth from the primary replica 2315 of FIG. 23 . At block 4215, the queue processing time calculator 2615 of FIG. 26 may determine the time required 3010 of FIG. 30 to process one I/O request. At block 4220, the queue processing time calculator 2615 of FIG. 26 multiplies the queue depth 3005 of FIG. 30 by the time taken 3010 of FIG. 30 to process one I/O request. A queue processing time 3015 of can be calculated.

43 is a flowchart illustrating a procedure for estimating a required time required to process the I/O request 905 of FIG. 23 according to an embodiment of the present invention. 43 may show a procedure used by the local prediction analyzer 2430 of FIG. 24 or the remote prediction analyzer 2435 of FIG. 24 . Any reference to the local prediction analyzer 2430 of FIG. 24 is intended to also refer to the remote prediction analyzer 2435 of FIG. 24 .

At block 4305 , the local prediction analyzer 2430 of FIG. 24 may access historical information from the database 2425 of FIG. 24 . At block 4310 , the local prediction analyzer 2430 of FIG. 24 may estimate the required time based on previous information in the database 2425 of FIG. 24 . The local prediction analyzer 2430 of FIG. 24 may use any desired approach for estimating the lead time. Exemplary approaches include calculating the mean, median, and mode of the prior information, applying weighted functions to the prior information, and performing a linear regression analysis. Embodiments of the present invention may also apply other approaches to estimating the required time.

44 shows weights 2635, 2640, and 2645 of FIG. 26, weights 2735, 2740 and 2745 of FIG. 27, weight 3310 of FIG. 33, and FIG. 37, according to an embodiment of the present invention. A flow chart showing the procedure for using linear regression analysis to determine the weights 3715 of . At block 4405 , weight generator 2630 of FIG. 26 and weight generator 2730 of FIG. 27 may determine a sliding window to use for previous information in database 2425 of FIG. 24 . At block 4410 , weight generator 2630 of FIG. 26 and weight generator 2730 of FIG. 27 may generate weights using linear regression analysis through windows in database 2425 of FIG. 24 .

45 is a flowchart illustrating a procedure for calculating the remote estimated required time 3710 of FIG. 37 by the remote estimated required time calculator 2725 of FIG. 27 according to an embodiment of the present invention. At block 4505 , the communication time calculator 2705 of FIG. 27 may calculate the communication time 3410 of FIG. 34 . At block 4510 , the remote processor time calculator 2710 of FIG. 27 may calculate the remote processor time 3515 of FIG. 35 . At block 4515 , the remote garbage collection time calculator 2715 of FIG. 27 may calculate the remote garbage collection time. At block 4520 , the teleprediction analyzer 2435 of FIG. 24 may calculate the expected remote time 3605 of FIG. 36 . At block 4525 , the weight generator 2730 of FIG. 27 may calculate weights applied at various times to be used in calculating the remote estimated time required 3710 of FIG. 37 . Finally, in block 4530 , the remote estimated time required calculator 2725 of FIG. 27 may calculate the remote estimated time required 3710 of FIG. 37 from all this information.

46 shows a procedure for the communication time calculator 2705 of FIG. 27 to determine the communication time 3410 of FIG. 34 with the secondary replicas 2320 and 2325 of FIG. 23, according to an embodiment of the present invention. is a flow chart. At block 4605 of FIG. 46 , the ping logic 3405 of FIG. 34 may check the communication state of each of the secondary replicas 2320 and 2325 of FIG. 23 . Since there may be more than one secondary replica, dashed arrow 4610 may be repeated as many times as block 4605 is needed. Alternatively, at block 4615 , the communication time calculator 2705 of FIG. 27 calculates the communication time 3410 of FIG. 34 from previous information stored in the database 2425 of FIG. 24 to calculate the communication time 3410 of FIG. ) can be calculated. The communication time calculator 2705 of FIG. 27 calculates the mean, median, and mode of the prior information in the database 2425 of FIG. 24 or a linear regression analysis on the prior information in the database 2425 of FIG. Any preferred approach for calculating the communication time 3410 of FIG. 34 may be used, including performing Alternatively, the communication time calculator 2705 of FIG. 27 may access a storage graph of information regarding the layout of various nodes and distances, delays, and bandwidths between nodes. From this information, the communication time calculator 2705 of FIG. 27 can calculate the communication time 3410 of FIG. 34 .

47 is a flowchart illustrating a procedure for the remote processor time calculator 2710 of FIG. 27 to determine the remote processor time 3515 of FIG. 35 according to an embodiment of the present invention. At block 4705 of FIG. 47 , the query logic 2415 of FIG. 24 may interrogate the processor loads 3505 of FIG. 35 to auxiliary replicas 2320 and 2325 of FIG. 23 . Because there may be more than one secondary replica, dashed arrow 4710 shows that block 4705 may be repeated as many times as necessary to query all secondary replicas. At block 4715 , the receive logic 2420 of FIG. 24 may receive the remote processor load(s) from the secondary replicas.

At block 4720 , the query logic 2415 of FIG. 24 may query the secondary replicas 2320 and 2325 of FIG. 23 for the software stack loads 3510 of FIG. 35 . Because there may be more than one secondary replica, dashed arrow 4725 shows that block 4720 may be repeated as many times as necessary to query all secondary replicas. At block 4730 , the receive logic 2420 of FIG. 24 may receive the remote software stack load(s) from the secondary replicas.

Finally, at block 4735 , the remote processor time calculator 2710 of FIG. 27 calculates the remote processor load(s) 3505 and remote software stack load(s) 3510 of FIG. 35 into the remote processor time of FIG. 35 . (3515) can be mapped.

15A-22B and 39A-47 , some embodiments of the present invention are shown. However, those skilled in the art will recognize other embodiments of the present invention by changing the order of blocks, omitting blocks, or including connections not shown in the drawings. All such variations of flowcharts, whether explicitly described or not, are considered embodiments of the present invention.

The following description is intended to provide a brief and general description of a suitable machine or machines in which some aspects of the inventive subject matter may be implemented. The machine or machines are at least in part by instructions received from another machine, interaction with a virtual reality (VR) environment, biofeedback, or other input signals as well as input from conventional input devices such as a keyboard, mouse, etc. can be controlled. As used herein, the term “machine” is intended to broadly encompass a single machine, a virtual machine, or a system coupled to communicate with machines, virtual machines, or devices operating together. Exemplary machines include personal computers, workstations, servers, portable computers, handheld devices, phones, tablets, as well as transport devices such as personal or public transport such as, for example, cars, trains, taxis, etc. computing devices, such as

The machine or machines may include embedded controllers such as programmable or non-programmable logic devices or arrays, application specific integrated circuits (ASICs), embedded computers, smart cards, and the like. The machine or machines may utilize one or more connections to one or more remote machines, such as through a network interface, modem, or other communication couplings. Machines may be connected to each other by means of physical and/or logical networks such as intranets, the Internet, local area networks (LANs), wide area networks (WANs), and the like. Those skilled in the art will appreciate that network communication utilizes a variety of wired and/or wireless near or far carriers and protocols including radio frequency (RF), satellite, microwave, IEEE 802.11, Bluetooth, optical, infrared, cable, laser, etc. will understand that

Embodiments of the inventive concept are functions, procedures, data structures, applications that, when accessed by a machine, cause a machine to perform tasks or define abstract data types or low-level hardware contexts. It may be described with reference to or cooperatively with associated data, including programs and the like. Associated data may include, for example, volatile and/or non-volatile memory such as RAM, ROM, etc., or other storage devices, and hard drives, floppy disks, optical storage, tapes, flash memory, memory sticks, digital video disks, etc. , may be stored in an associated storage medium including bio-storage and the like. The associated data may be transmitted in the form of packets, serial data, parallel data, transmission signals, etc., via transmission environments including physical and/or logical networks, and may be used in a compressed or encrypted format. Associated data may be used in a distributed environment and may be stored locally and/or remotely for machine access.

Embodiments of the inventive concept are tangible, non-transitory machine-readable media that are executable by one or more processors and contain instructions for performing the elements of the inventive concept as described herein. may include

Since the principles of the present invention have been described and illustrated with reference to the illustrated embodiments, the illustrated embodiments may be modified in order and in detail, and may be combined in any desired manner, without departing from these principles. This will be understood. Although the preceding discussion has focused on specific embodiments, other configurations are also contemplated. Specifically, although descriptions such as "according to an embodiment of the technical idea of the present invention" or similar expressions are used in this specification, these phrases generally refer to possibilities of the embodiment, and refer to the technical spirit of the present invention in the specific embodiment. It is not intended to be limited to configurations. As used herein, these terms may refer to the same or different embodiments, combinable into other embodiments.

The above-described embodiments are not to be construed as limiting the technical spirit of the present invention thereto. Although a small number of embodiments have been described, those skilled in the art will fully appreciate that many modifications are possible to these embodiments without departing substantially from the novel descriptions and advantages of the present disclosure. Accordingly, all such modifications are intended to be included within the scope of the present invention as defined in the claims.

Embodiments of the present invention may be extended to the following descriptions without limitation:

Description 1. An embodiment of the present invention,

at least one storage device (140, 145, 150, 155, 160, 165, 225, 230) comprising a primary replica (2315) of data;

Local estimated time taken 3305 to complete input/output (I/O) request 905 on primary replica 2315 and I/O request 905 on at least one of secondary replicas 2320 , 2325 of data ) a cost analyzer 2310 for calculating a remote estimated time required to complete 3710; and

I for sending an I/O request 905 to one of the primary replica 2315 and the at least one secondary replica 2320 , 2325 in response to the local estimated time spent 3305 and the at least one remote estimated time spent 3710 . Distributed storage system nodes 124 , 130 , 135 including /O relocator 215 .

Description 2. An embodiment of the present invention includes the distributed storage system nodes 125, 130, 135 according to Description 1, and includes at least one storage device 140, 145, 150, 155, 160, 165, 225, 230. includes solid state drives (SSDs) 140 , 145 , 150 , 155 , 160 , 165 , 225 , 230 .

Description 3. An embodiment of the present invention includes the distributed storage system nodes 125, 130, and 135 according to the description 1, wherein the distributed storage system nodes 125, 130 and 135 are network attached solid state drives (Network Attached SSD). and an Ethernet SSD.

Description 4. An embodiment of the present invention includes the distributed storage system nodes 125 , 130 , 135 according to Description 1 , and includes at least one storage device 140 , 145 , 150 , 155 , 160 , 165 , 225 , 230 . Only when is currently performing garbage collection, the I/O relocation 215 operates to relocate the I/O request 905 .

Description 5. An embodiment of the present invention includes the distributed storage system nodes 125, 130, 135 according to description 4,

The cost analyzer 2310 includes a local time estimator 2405 for calculating a local estimated time taken 3305 for processing an I/O request 905 at the primary replica 2315 ;

I/O relocation 215 is

storage 2505 for threshold time 2525; and

and a first comparator for comparing the local estimated required time 3305 with a threshold time.

Description 6. An embodiment of the present invention includes the distributed storage system nodes 125, 130, 135 according to description 5,

If the local estimated turnaround time 3305 is less than the threshold time, the I/O relocation 215 operates to direct the I/O request 905 to the primary replica 2315 .

Description 7. An embodiment of the present invention includes the distributed storage system nodes 125, 130, 135 according to description 5, wherein the local time estimator 2405 is

a local garbage collection time calculator 2605 for calculating a local garbage collection time 2820;

a local estimated garbage collection time calculator 2610 for calculating a local estimated garbage collection time 2905;

storage 2620 for local garbage collection weights 2636 and expected garbage collection weights 2640; and a local estimated time taken to compute a local estimated time taken 3305 from the local garbage collection time 2820 , the local estimated garbage collection time 2905 , the local garbage collection weight 2635 , and the expected garbage collection weight 2640 . A calculator 2625 is included.

Description 8. An embodiment of the present invention includes the distributed storage system nodes 125 , 130 , and 135 according to the description 7 , wherein the local estimation time required calculator 2625 sets the local garbage collection weight to the local garbage collection time 2820 . and calculate the local estimated required time 3305 as the sum of the product multiplied by 2635 and the local estimated garbage collection time 2905 multiplied by the expected garbage collection weight 2640 .

Description 9. An embodiment of the present invention includes the distributed storage system nodes 125 , 130 , and 135 according to the description 8 , wherein the local estimation time required calculator 2625 sets the local garbage collection weight to the local garbage collection time 2820 . (2635) times the local estimated garbage collection time (2905) multiplied by the expected garbage collection weight (2640), and the queuing time (3015) multiplied by the queuing weight (2645). operable to calculate time 3305 .

Description 10. An embodiment of the present invention includes the distributed storage system nodes 125, 130, 135 according to description 7,

The cost analyzer 2310 is

query logic to query the primary replica 2315 for the actual number of empty pages 2805; and

further comprising receiving logic (2420) for receiving the actual number of empty pages (2805) from the primary replica (2315);

The local garbage collection time calculator 2605 calculates a difference value by subtracting the actual number of empty pages 2805 from a threshold value 2810 of the number of empty pages for the primary replica 2315, and calculates the difference value as the local average garbage It operates to multiply the collection time 2815 to calculate a local garbage collection time 2820 .

Description 11. An embodiment of the present invention includes a distributed storage system node 125, 130, 135 according to statement 10, wherein the local garbage collection time calculator 2605 determines the delay associated with programmable valid pages within each erase block. 2825) is added to the local garbage collection time 2820.

Description 12. An embodiment of the present invention includes the distributed storage system nodes 125 , 130 , 135 according to description 10 , wherein the query logic 2415 periodically places the actual number of empty pages in the primary replica 2315 ( 2805 ). works to inquire about

Description 13. An embodiment of the present invention includes the distributed storage system nodes 125, 130, 135 according to description 7,

The cost analyzer 2310 is

query logic to query the primary replica 2315 for the actual number of empty pages 2805; and

further comprising receiving logic (2420) for receiving the actual number of empty pages (2805) from the primary replica (2315);

The local expected garbage collection time calculator 2610 computes a difference value by subtracting the actual number of empty pages 2805 from a threshold value of the number of empty pages for the primary replica 2315 ( 2810 ), and the difference value as a local average It operates to multiply the garbage collection time 2815 to calculate a local expected garbage collection time 2905 .

Description 14. An embodiment of the present invention includes a distributed storage system node 125, 130, 135 according to statement 13, wherein the local expected garbage collection time calculator 2610 is a delay associated with programmable valid pages within each erase block. (2825) is added to the local garbage collection time (2820).

Description 15. An embodiment of the present invention includes the distributed storage system nodes 125 , 130 , 135 according to description 13 , wherein the query logic 2415 periodically places the actual number of empty pages in the primary replica 2315 ( 2805 ). works to inquire about

Description 16. An embodiment of the present invention includes the distributed storage system nodes 125, 130, 135 according to description 7,

local time estimator 2405 includes queue processing time calculator 2615 for calculating queue processing time 3015;

storage 2620 includes storage for queue processing weights 2645 ;

The local estimated time spent calculator 2625 includes a local garbage collection time 2820, a local estimated garbage collection time 2905, a queue processing time 3015, a local garbage collection weight 2635, an estimated garbage collection weight 2640, and It may operate to calculate a local estimated time taken from the queue processing weight 2645 .

Description 17. An embodiment of the present invention comprises the distributed storage system nodes 125, 130, 135 according to description 16,

The cost analyzer 2310 is

query logic for querying the primary replica 2315 for the number 3005 of I/O requests 905 pending on the primary replica 2315; and

receiving logic for receiving from the primary replica (2315) a number (3005) of I/O requests (905) pending on the primary replica (2315);

The queue processing time calculator 2615 multiplies the number of I/O requests 905 pending on the primary replica 2315 ( 3005 ) by the time required to process one I/O request ( 905 ) 3010 to create a queue operable to calculate processing time 3015 .

Description 18. An embodiment of the present invention includes the distributed storage system nodes 125 , 130 , 135 according to statement 17 , wherein the query logic 2415 is periodically pending on the primary replica 2315 on the primary replica 2315 . It operates to query the number 3005 of I/O requests 905 .

Description 19. An embodiment of the present invention includes the distributed storage system nodes 125, 130, 135 according to description 7, wherein the cost analyzer 2310 comprises:

Previous local garbage collection information 3105 on the primary replica 2315, worst case estimated local garbage collection 3110 on the primary 2315, average case estimated local garbage collection 3115 on the primary 2315, a database 2425 storing information including at least one of prior processing time information 3120 for the primary replica 2315, and an average case estimated processing time 3130 on the primary replica 2315; and

It further includes a local prediction analyzer 2430 for calculating an expected local time for the primary replica 2315 from information stored in the database 2425 .

Description 20. An embodiment of the present invention includes the distributed storage system nodes 125 , 130 , 135 according to description 19 , wherein the local estimated time required calculator 2625 includes: a local garbage collection time 2820 , a local estimated garbage collection It operates to calculate a local estimated time taken 3305 from the time 2905 , the expected local time 3205 , the local garbage collection weight 2635 , and the expected garbage collection weight 2640 .

Description 21. An embodiment of the present invention includes the distributed storage system nodes 125, 130, 135 according to description 7, wherein the local time estimator 2405 includes a local garbage collection weight 2635 and an expected garbage collection weight 2640 It further includes a weight generator 2630 for generating .

Description 22. An embodiment of the present invention comprises distributed storage system nodes 125, 130, 135 according to statement 21, wherein the weight generator 2630 performs a linear regression analysis based on previous data on the primary replica 2315. used to generate a local garbage collection weight 2635 and an expected garbage collection weight 2640 .

Description 23. An embodiment of the present invention comprises a distributed storage system node 125 , 130 , 135 according to statement 22 , wherein the old data comes from the use of a sliding window of the primary replica 2315 .

Description 24. An embodiment of the present invention comprises the distributed storage system nodes 125, 130, 135 according to description 5,

The cost analyzer 2310 includes a remote time estimator ( 2410),

I/O relocation 215 is,

a second comparator 2515 for comparing the local estimated time spent 3305 with the at least one remote estimated time spent 3710; and

A primary replica 2315, and at least one secondary replica 2320, 2325 for handling I/O requests 905 with a minimum time from a local estimated time spent 3305 and at least one remote estimated time spent 3710 ) further includes a selector 2520 for selecting one.

Description 25. An embodiment of the present invention comprises the distributed storage system nodes 125, 130, 135 according to description 24,

The remote time estimator 2410 includes:

for calculating a communication time 3410 between distributed storage system nodes 125 , 130 , 135 and at least one secondary storage system node 125 , 130 , 135 comprising at least one secondary replica 2320 , 2325 . communication time calculator 2705;

a remote processor time calculator (2710) for calculating remote processor time (3515) for at least one secondary storage system node (125, 130, 135);

a remote garbage collection time calculator (2715) for calculating a remote garbage collection time (3705) for at least one secondary replica (2320, 2325);

storage 2720 for communication time weights 2735 , remote processor time weights 2740 , and remote garbage collection time weights; and

Remote estimated time taken from communication time 3410 , remote processor time 3515 , remote garbage collection time 3705 , communication time weight 2735 , remote processor time weight 2740 , and remote garbage collection time weight 2745 . and a remote estimated time required calculator 2725 for calculating 3710 .

Description 26. An embodiment of the present invention includes the distributed storage system nodes 125 , 130 , 135 according to description 25 , wherein the remote estimated time required calculator 2725 calculates the communication time weight 2735 to the communication time 3410 . Remote estimated time taken 3710 as the sum of the product, remote processor time 3515 multiplied by remote processor time weight 2740, and remote garbage collection time 3705 multiplied by remote garbage collection time weight 2745 ) can be operated to calculate

Description 27. An embodiment of the present invention includes the distributed storage system nodes 125 , 130 , 135 according to statement 25 , wherein the communication time calculator 2705 is at least one auxiliary storage system for measuring the communication time 3410 . and a ping logic 3405 for checking the communication status of the nodes 125, 130, and 135.

Description 28. An embodiment of the present invention includes the distributed storage system nodes 125 , 130 , 135 according to description 27 , wherein the ping logic 3405 periodically has at least one auxiliary storage for measuring the communication time 3410 . It operates to check the communication state of the system nodes (125, 130, 135).

Description 29. An embodiment of the present invention comprises the distributed storage system nodes 125, 130, 135 according to description 25,

The cost analyzer 2310 is

query logic 2415 for querying the at least one secondary storage system node (125, 130, 135) for a remote processor load (3505) on the at least one secondary storage system node (125, 130, 135); and

further comprising receiving logic (2420) for receiving a remote processor load (3505) from at least one secondary storage system node (125, 130, 135);

The remote processor time calculator 2710 is operative to calculate the remote processor time 3515 in response to the remote processor load 3505 .

Statement 30. An embodiment of the present invention includes a distributed storage system node (125, 130, 135) according to statement 29, wherein the query logic 2415 is remote to at least one secondary storage system node (125, 130, 135). It operates to periodically query the processor load 3505 .

Description 31. An embodiment of the present invention comprises the distributed storage system nodes 125, 130, 135 according to description 29,

query logic 2415 is operative to query at least one secondary storage system node 125 , 130 , 135 for a remote software stack load 3515 on at least one secondary storage system node 125 , 130 , 135 ;

receive logic 2420 is operative to receive remote software stack load 3510 from at least one secondary storage system node 125 , 130 , 135 ;

The remote processor time calculator 2710 is operative to calculate the remote processor time 3515 in response to the remote processor load 3505 and the remote software stack load 3510 .

Description 32. An embodiment of the present invention includes a distributed storage system node (125, 130, 135) according to statement 31, wherein the query logic 2415 is remote to at least one secondary storage system node (125, 130, 135). It operates to periodically query the software stack load 3510 .

Description 33. An embodiment of the present invention comprises the distributed storage system nodes 125, 130, 135 according to description 25,

The cost analyzer 2310 is

query logic 2415 for querying the actual number of empty pages 2805 to the at least one secondary replica 2320 , 2325 ; and

further comprising receiving logic (2420) for receiving the actual number of empty pages (2805) from the at least one secondary replica (2320, 2325);

The remote garbage collection time calculator 2715 calculates a difference value by subtracting the actual number of empty pages 2805 from a threshold value 2810 of the number of empty pages for the at least one secondary replica 2320 , 2325 , the difference and multiply the value by the remote average garbage collection time ( 4160 ) to calculate a remote garbage collection time ( 3705 ).

Statement 34. An embodiment of the present invention includes a distributed storage system node 125, 130, 135 according to statement 33, wherein the remote garbage collection time calculator 2715 computes a delay associated with the programmable valid pages of each erase block. 2825) is added to the remote garbage collection time 3705.

Description 35. An embodiment of the present invention includes the distributed storage system nodes 125, 130, 135 according to description 33, wherein the query logic 2415 is the number of actual empty pages in the at least one secondary replica 2320, 2325. (2805) operates to interrogate periodically.

Description 36. An embodiment of the present invention comprises the distributed storage system nodes 125, 130, 135 according to description 25, wherein the cost analyzer 2310 comprises:

previous communication time information 3135 for at least one secondary replica 2320, 2325, worst case estimate 3140 of communication time 3410 for at least one secondary replica 2320, 2325, at least one secondary Average case estimate 3145 of communication time 3410 for replicas 2320, 2325, previous remote processor time information 3150 for at least one secondary replica 2320, 2325, at least one secondary replica 2320 , 2325 , worst case estimate 3155 for remote processor time 3515 , average case estimate 3160 for remote processor time 3515 on at least one secondary replica 2320 , 2325 , at least one Previous remote garbage collection information 3165 for secondary replicas 2320, 2325, worst case estimate 3170 for remote garbage collection on at least one secondary replica 2320, 2325, and at least one secondary replica 2320 , 2325) a database 2425 for storing information including at least one of an average case estimate for remote garbage collection 3175; and

from information 3135 , 3140 , 3145 , 3150 , 3155 , 3160 , 3165 , 3170 , 3175 stored in database 2425 , for calculating an estimated remote time 3605 for at least one secondary replica 2320 , 2325 . A remote prediction analyzer 2435 may be further included.

Description 37. An embodiment of the present invention includes the distributed storage system nodes 125 , 130 , 135 according to statement 36 , wherein the remote estimated time spent calculator 2725 includes a communication time 3410 , a remote processor time 3515 , From the remote garbage collection time 3705 , the estimated remote time 3605 , the communication time weight 2735 , the remote processor time weight 2740 , and the remote garbage collection time weight 2745 , calculate a remote estimated time taken 3710 . act to do

Description 38. An embodiment of the present invention includes the distributed storage system nodes 125 , 130 , 135 according to statement 25 , wherein the remote time estimator 2410 comprises: a communication time weight 2735 , a remote processor time weight 2740 . , and a weight generator 2730 for generating the remote garbage collection time weight 2745 .

Description 39. An embodiment of the present invention comprises a distributed storage system node (125, 130, 135) according to statement 38, wherein the weight generator (2730) is linear based on previous data of the at least one secondary replica (2320, 2325). The regression analysis is used to generate communication time weights 2735 , remote processor time weights 2740 , and remote garbage collection time weights 2745 .

Statement 40. An embodiment of the present invention comprises a distributed storage system node (125, 130, 135) according to statement 39, wherein the old data comes from the use of sliding windows of at least one secondary replica (2320, 2325).

Description 41. An embodiment of the present invention,

Calculate a local estimated time taken 3305 to process an input/output (I/O) request 905 on the primary replica 2315 of the data, the primary replica 2315 being the storage devices 140 , 145 , 150 , a local time estimator 2405 included in 155, 160, 165, 225, 230; and

A cost analyzer comprising a remote time estimator 2410 for calculating at least one remote estimated time taken 3710 to process an I/O request 905 on at least one secondary replica 2320 , 2325 of data. including,

The cost analyzer 2310 determines that the I/O relocator 215 converts the I/O request 905 to the primary replica, in response to the local estimated time spent 3305 and at least one remote estimated time spent 3710 . 2315) and at least one of the secondary replicas (2320, 2325).

Statement 42. An embodiment of the present invention includes a cost analyzer 2310 according to statement 41, wherein the storage devices 140, 145, 150, 155, 160, 165, 225, 230 are solid state drives 140, 145, 150, 155, 160, 165, 225, 230 (SSD).

Description 43. An embodiment of the present invention includes a cost analyzer 2310 according to statement 41, wherein the cost analyzer 2310 is only activated while the primary replica 2315 is performing garbage collection.

Description 44. An embodiment of the present invention comprises the cost analyzer 2310 according to statement 43, wherein the local time estimator 2405 comprises:

a local garbage collection time calculator 2605 for calculating a local garbage collection time 2820;

a local estimated garbage collection time calculator 2610 for calculating a local estimated garbage collection time 2905;

storage 2620 for local garbage collection weights 2635 and expected garbage collection weights 2640; and

Local estimated time taken to compute a local estimated time taken 3305 from the local garbage collection time 2820 , the local estimated garbage collection time 2905 , the local garbage collection weights 2635 , and the expected garbage collection weights 2640 . A calculator 2625 is included.

Description 45. An embodiment of the present invention includes the cost analyzer 2310 according to statement 44, wherein the local estimated time taken calculator 2625 multiplies the local garbage collection time 2820 by the local garbage collection weight 2635 and It operates to calculate the local estimated required time 3305 as the sum of the local estimated garbage collection time 2905 multiplied by the expected garbage collection weight 2640 .

Description 46. An embodiment of the present invention includes the cost analyzer 2310 according to description 45, wherein the local estimated time required calculator 2625 multiplies the local garbage collection time 2820 by a local garbage collection weight, It operates to calculate a local estimated time required 3305 as the sum of the collection time 2905 times the expected garbage collection weight 2640 and the queue processing time 3015 times the queue processing weight 2645 .

Description 47. An embodiment of the present invention comprises a cost analyzer 2310 according to statement 44,

The cost analyzer 2310 includes query logic 2415 for querying the primary replica 2315 for the actual number of empty pages 2805; and

receiving logic (2420) for receiving the actual number of empty pages (2805) from the primary replica (2315);

The local garbage collection time calculator 2605 calculates a difference value by subtracting the actual number of empty pages 2805 from a threshold value of the number of empty pages 2810, and multiplies the difference value by the local average garbage collection time 2815 to obtain the local It operates to calculate a garbage collection time 2820 .

Statement 48. An embodiment of the present invention includes a cost analyzer 2310 according to statement 47, wherein the local garbage collection time calculator 2605 calculates a delay 2825 associated with programmable valid pages within each erase block for local garbage collection. Add operation to time 2820 .

Statement 49. An embodiment of the present invention includes a cost analyzer 2310 according to statement 47, wherein the query logic 2415 is configured to periodically query the at least one primary replica 2315 for the number of actual empty pages 2805. It works.

Statement 50. An embodiment of the present invention comprises a cost analyzer 2310 according to statement 44,

The cost analyzer 2310 is

query logic 2415 for querying the primary replica 2315 for the number of actual empty pages 2805; and

receiving logic (2420) for receiving the actual number of empty pages (2805) from the primary replica (2315);

The local estimated garbage collection time calculator 2610 calculates a difference value by subtracting the actual number of empty pages from the threshold value 2810 of the number of empty pages for the primary replica 2315, and calculates the difference value as the local average garbage collection time ( 2815) to calculate the local expected garbage collection time 2905.

Description 51. An embodiment of the present invention includes a cost analyzer 2310 according to statement 50, wherein the local estimated garbage collection time calculator 2610 calculates a local estimate of a delay 2825 associated with programmable valid pages within each erase block. Add operation to garbage collection time 2905.

Statement 52. An embodiment of the present invention includes a cost analyzer 2310 according to statement 50, wherein the query logic 2415 is configured to periodically query the at least one primary replica 2315 for the number of actual empty pages 2805. It works.

Description 53. An embodiment of the present invention comprises a cost analyzer 2310 according to statement 44,

local time estimator 2405 includes queue processing time calculator 2615 for calculating queue processing time 3015;

Storage 2630 includes storage for queue processing weights 2645,

The local estimated time taken calculator 2625 includes a local garbage collection time 2820, a local estimated garbage collection time 2905, a queue processing time 3015, a local garbage collection weight 2635, an estimated garbage collection weight 2640, and calculate a local estimated time taken 3305 from the queue processing weight 2645 .

Description 54. An embodiment of the present invention comprises a cost analyzer 2310 according to statement 53,

The cost analyzer 2310 is

query logic for querying the primary replica 2315 for the number 3005 of I/O requests 905 pending on the primary replica 2315; and

receiving logic for receiving from the primary replica (2315) a number (3005) of I/O requests (905) pending on the primary replica (2315);

Queue processing time calculator 2615 multiplies the number of I/O requests 905 pending on the primary replica 2315 ( 3005 ) by the time taken to process one I/O request ( 905 ) 3010 to process the queue. operable to calculate time 3015 .

Description 55. An embodiment of the present invention includes a cost analyzer 2310 according to statement 54, wherein the query logic 2415 notes the number 3005 of pending I/O requests 905 to the primary replica 2315. It operates to periodically inquire the replica 2315.

Statement 56. An embodiment of the present invention comprises a cost analyzer 2310 according to statement 44,

Previous local garbage collection information 3105 on the primary replica 2315, worst case estimated local garbage collection 3110 on the primary 2315, average case estimated local garbage collection 3115 on the primary 2315, a database 2425 storing information including at least one of prior processing time information 3120 for the primary replica 2315, and an average case estimated processing time 3130 on the primary replica 2315; and

It further includes a local prediction analyzer 2430 for calculating an expected local time for the primary replica 2315 from information stored in the database 2425 .

Description 57. An embodiment of the present invention includes the cost analyzer 2310 according to statement 56, wherein the local estimated time taken calculator 2625 includes: a local garbage collection time 2820, a local estimated garbage collection time 2905, an estimated It operates to calculate a local estimated time taken 3305 from the local time 3205 , the local garbage collection weight 2635 , and the expected garbage collection weight 2640 .

Description 58. An embodiment of the present invention includes a cost analyzer 2310 according to statement 44, wherein the local time estimator 2405 is a weight generator for generating a local garbage collection weight 2635 and an expected garbage collection weight 2640. (2630).

Statement 59. An embodiment of the present invention includes a cost analyzer 2310 according to statement 58, wherein the weight generator 2630 uses a linear regression analysis based on previous data for the primary replica 2315 for local garbage collection. It operates to generate a weight 2635 and an expected garbage collection weight 2640 .

Description 60. An embodiment of the present invention comprises a cost analyzer 2310 according to statement 59, wherein the previous data comes from a sliding window of use of the primary replica 2315.

Description 61. An embodiment of the present invention comprises a cost analyzer 2310 according to statement 43, wherein the remote time estimator 2410 comprises:

for calculating a communication time 3410 between distributed storage system nodes 125 , 130 , 135 and at least one secondary storage system node 125 , 130 , 135 comprising at least one secondary replica 2320 , 2325 . communication time calculator 2705;

a remote processor time calculator (2710) for calculating remote processor time (3515) for at least one secondary storage system node (125, 130, 135);

a remote garbage collection time calculator (2715) for calculating a remote garbage collection time (3705) for at least one secondary replica (2320, 2325);

storage 2720 for communication time weights 2735 , remote processor time weights 2740 , and remote garbage collection time weights; and

Remote estimated time taken from communication time 3410 , remote processor time 3515 , remote garbage collection time 3705 , communication time weight 2735 , remote processor time weight 2740 , and remote garbage collection time weight 2745 . and a remote estimated time required calculator 2725 for calculating 3710.

Description 62. An embodiment of the present invention includes the cost analyzer 2310 according to statement 61, wherein the remote estimated time required calculator 2725 is a communication time 3410 multiplied by a communication time weight 2735, remote processor time. calculate remote estimated time taken 3710 as the sum of 3515 multiplied by remote processor time weight 2740, and remote garbage collection time 3705 multiplied by remote garbage collection time weight 2745. can

Description 63. An embodiment of the present invention comprises a cost analyzer 2310 according to statement 61 , wherein the communication time calculator 2705 comprises at least one auxiliary storage system node 125 , 130 , for measuring the communication time 3410 , 135) and includes a ping logic 3405 for checking the communication state.

Description 64. An embodiment of the present invention includes a cost analyzer 2310 according to statement 63, wherein the ping logic 3405 is configured to periodically measure the communication time 3410 at least one secondary storage system node 125, 130 , 135) to check the communication status.

Description 65. An embodiment of the present invention comprises a cost analyzer 2310 according to statement 61,

The cost analyzer 2310 is

query logic 2415 for querying the at least one secondary storage system node (125, 130, 135) for a remote processor load (3505) on the at least one secondary storage system node (125, 130, 135); and

further comprising receiving logic (2420) for receiving a remote processor load (3505) from the at least one secondary storage system node (125, 130, 135);

The remote processor time calculator 2710 is operative to calculate the remote processor time 3515 in response to the remote processor load 3505 .

Statement 66. An embodiment of the present invention includes a cost analyzer 2310 according to statement 65, wherein the query logic 2415 assigns the remote processor load 3505 to the at least one secondary storage system node 125, 130, 135. Operates to inquire periodically

Description 67. An embodiment of the present invention comprises a cost analyzer 2310 according to statement 65,

query logic 2415 is operative to query at least one secondary storage system node 125 , 130 , 135 for a remote software stack load 3515 on at least one secondary storage system node 125 , 130 , 135 ;

receive logic 2420 is operative to receive remote software stack load 3510 from at least one secondary storage system node 125 , 130 , 135 ;

The remote processor time calculator 2710 is operative to calculate the remote processor time 3515 in response to the remote processor load 3505 and the remote software stack load 3510 .

Description 68. An embodiment of the present invention includes a cost analyzer 2310 according to statement 67, wherein the query logic 2415 loads the remote software stack 3510 onto the at least one secondary storage system node 125, 130, 135. It operates to inquire periodically.

Description 69. An embodiment of the present invention comprises a cost analyzer 2310 according to statement 61,

The cost analyzer 2310 is

query logic 2415 for querying the actual number of empty pages 2805 to the at least one secondary replica 2320 , 2325 ; and

further comprising receiving logic (2420) for receiving the actual number of empty pages (2805) from the at least one secondary replica (2320, 2325);

The remote garbage collection time calculator 2715 calculates a difference value by subtracting the actual number of empty pages 2805 from a threshold value 2810 of the number of empty pages for the at least one secondary replica 2320 , 2325 , the difference and multiply the value by the remote average garbage collection time ( 4160 ) to calculate a remote garbage collection time ( 3705 ).

Statement 70. An embodiment of the present invention includes a cost analyzer 2310 according to statement 69, wherein the remote garbage collection time calculator 2715 calculates a delay 2825 associated with programmable valid pages of each erase block for remote garbage collection. Add operation to time 3705 .

Description 71. An embodiment of the present invention includes a cost analyzer 2310 according to statement 69, wherein the query logic 2415 periodically records the number of actual empty pages 2805 in the at least one secondary replica 2320, 2325. act to inquire.

Description 72. An embodiment of the present invention comprises a cost analyzer 2310 according to statement 61,

previous communication time information 3135 for at least one secondary replica 2320, 2325, worst case estimate 3140 of communication time 3410 for at least one secondary replica 2320, 2325, at least one secondary Average case estimate 3145 of communication time 3410 for replicas 2320, 2325, previous remote processor time information 3150 for at least one secondary replica 2320, 2325, at least one secondary replica 2320 , 2325 , worst case estimate 3155 for remote processor time 3515 , average case estimate 3160 for remote processor time 3515 on at least one secondary replica 2320 , 2325 , at least one Previous remote garbage collection information 3165 for secondary replicas 2320, 2325, worst case estimate 3170 for remote garbage collection on at least one secondary replica 2320, 2325, and at least one secondary replica 2320 , 2325) a database 2425 for storing information including at least one of an average case estimate for remote garbage collection 3175; and

from information 3135 , 3140 , 3145 , 3150 , 3155 , 3160 , 3165 , 3170 , 3175 stored in database 2425 , for calculating an estimated remote time 3605 for at least one secondary replica 2320 , 2325 . A remote prediction analyzer 2435 may be further included.

Statement 73. An embodiment of the present invention includes a cost analyzer 2310 according to statement 72, wherein the remote estimated time spent calculator 2725 includes a communication time 3410, a remote processor time 3515, and a remote garbage collection time 3705. ) , the estimated remote time 3605 , the communication time weight 2735 , the remote processor time weight 2740 , and the remote garbage collection time weight 2745 , to calculate the remote estimated time taken 3710 .

Statement 74. An embodiment of the present invention includes a cost analyzer 2310 according to statement 61, wherein the remote time estimator 2410 includes: a communication time weight 2735, a remote processor time weight 2740, and a remote garbage collection time It may further include a weight generator 2730 for generating the weight 2745 .

Statement 75. An embodiment of the present invention comprises a cost analyzer 2310 according to statement 74, wherein the weight generator 2730 uses a linear regression analysis based on previous data of at least one secondary replica 2320, 2325, operates to generate a communication time weight 2735 , a remote processor time weight 2740 , and a remote garbage collection time weight 2745 .

Statement 76. An embodiment of the present invention comprises a cost analyzer 2310 according to statement 75, wherein the previous data comes from the use of sliding windows of at least one secondary replica 2320 , 2325 .

Description 77. An embodiment of the present invention,

In the distributed storage system nodes (125, 130, 135), an input/output (I/O) request (905), I requesting data from the primary replica (2315) in the distributed storage system nodes (125, 130, 135) receiving 3905 a primary replica 2315 comprising an /O request 905, storage 140, 145, 150, 155, 160, 165, 225, 230;

calculating ( 3920 ) a local estimated time taken ( 3305 ) to complete an I/O request ( 905 );

calculating ( 3940 ) at least one remote estimated time required ( 3710 ) for at least one secondary replica ( 2320 , 2325 ) storing the requested data;

comparing (3950) the local estimated time taken (3305) to the at least one remote estimated time taken (3710);

selecting (3955) one of the primary replica (2315) and the at least one secondary replica (2320, 2325) in response to the local estimated time spent (3305) and the at least one remote estimated time spent (3710);

and sending an I/O request (905) to a selected one of a primary replica (2315) and at least one secondary replica (2320, 2325).

Statement 78. An embodiment of the present invention comprises the method according to statement 77, wherein receiving ( 3905 ) an I/O request ( 905 ) at a distributed storage system node ( 125 , 130 , 135 ) comprises: I/O request 905 at (125, 130, 135), I/O request 905 requesting data from primary replica 2315 at distributed storage system nodes 125, 130, 135, solid state drive and receiving 3905 a primary replica 2315 comprising (SSD) 140 , 145 , 150 , 155 , 160 , 165 , 225 , 230 .

Statement 79. An embodiment of the present invention comprises the method according to statement 77, wherein the distributed storage system nodes 125, 130, 135 emerge from a set comprising a network-attached solid state drive (SSD) and an Ethernet SSD.

Statement 80. An embodiment of the invention includes the method according to statement 77, further comprising performing 3910 the method only while the primary replica 2315 is performing garbage collection.

Description 81. An embodiment of the invention comprises a method according to statement 80,

comparing 3925 the local estimated time taken 3305 to a threshold time 2525; and

processing 3915 the I/O request 905 at (2315) the primary replica if the local estimated turnaround time 3305 is less than the threshold time 2525.

Statement 82. An embodiment of the present invention includes the method according to statement 81, wherein processing 3915 an I/O request 905 at the primary replica 2315 comprises at least one of storing the requested data. calculating ( 3940 ) at least one remote estimated time spent ( 3710 ) for the secondary replicas ( 2320 , 2325 ), and comparing the local estimated time spent ( 3305 ) to the at least one remote estimated time spent ( 3710 ). and processing 3915 the I/O request 905 at the primary replica 2315 without 3950 .

Description 83. An embodiment of the present invention comprises the method according to statement 80, wherein calculating ( 3920 ) a local estimated time taken 3305 to complete an I/O request ( 905 ) comprises:

calculating (4005) a local garbage collection time (2820);

calculating 4010 a local expected garbage collection time 2905;

calculating 4030 a local estimated time taken 3305 from a local garbage collection time 2820, a local budget garbage collection time 2905, a local garbage collection weight 2635, and an expected garbage collection weight 2640 do.

Description 84. An embodiment of the present invention includes the method according to statement 83, wherein the step of calculating 4030 the estimated local estimated time taken 3305 includes: adding a local garbage collection weight 2635 to a local garbage collection time 2820 and calculating 4030 the local estimated required time 3305 as the sum of the multiplied value and the local estimated garbage collection time 2905 multiplied by the expected garbage collection weight 2640 .

Description 85. An embodiment of the present invention includes the method according to statement 84, wherein the step 4030 of calculating a local estimated time taken 3305 includes: a local garbage collection weight 2635 equal to a local garbage collection time 2820 Local estimated time taken 3305 as the sum of the product multiplied by the local estimated garbage collection time 2905 times the expected garbage collection weight 2640, and the queue processing time 3015 multiplied by the queuing weight 2645. The method further includes calculating 4030 .

Statement 86. An embodiment of the invention comprises the method according to statement 83, wherein calculating a local garbage collection time 2820 (4005) comprises:

determining ( 4105 ) whether the primary replica 2315 is performing garbage collection; and

and calculating 4005 a local garbage collection time 2820 if the primary replica 2315 is currently performing garbage collection.

Statement 87. An embodiment of the invention comprises the method according to statement 86, wherein calculating a local garbage collection time 2820 comprises:

querying the primary replica 2315 for the actual number of empty pages 2805;

calculating ( 4105 ) a difference value of the number of empty pages threshold ( 2810 ) for the primary replica ( 2315 ) minus the actual number of empty pages ( 2805 );

The method further includes multiplying the difference value by a local average garbage collection time 2815 to determine a local garbage collection time 2820 .

Statement 88. An embodiment of the present invention includes the method according to statement 87, wherein calculating a local garbage collection time 2820 comprises 4005 adding a delay 2820 associated with programmable valid pages within each erase block. Step 4155 is further included.

Statement 89. An embodiment of the present invention includes the method according to statement 87, further comprising periodically querying the primary replica 2315 for the number of actual empty pages 2805 (4120, 4125).

Statement 90. An embodiment of the present invention includes the method according to statement 86, wherein calculating a local garbage collection time 2820 (4005) comprises: previous local garbage collection information (3105) for a primary replica (2315); Local garbage collection time 2820 using at least one of a worst case estimate for local garbage collection 3110 on primary replica 2315, average case estimate for local garbage collection 3115 on primary replica 2315 It further includes the step of calculating (4110, 4115).

Description 91. An embodiment of the present invention comprises the method according to statement 83, wherein calculating 4010 a local expected garbage collection time 2905 comprises:

determining whether the primary replica 2315 is expected to begin garbage collection soon; and

and calculating 4010 an expected garbage collection time 2820 only if the primary replica is about to perform garbage collection soon.

Description 92. An embodiment of the invention comprises the method according to statement 91, wherein calculating 4010 a local expected garbage collection time 2820 comprises:

querying 4120 the number of actual blank pages 2820 to the primary replica 2315;

calculating ( 4150 ) a difference value of the number of empty pages threshold ( 2810 ) for the primary replica ( 2315 ) minus the actual number of empty pages ( 2805 ); and

The method further includes the step of multiplying the difference value by the local average garbage collection time 2815 ( 4160 ) to determine the local estimated time required ( 3305 ).

Statement 93. An embodiment of the present invention includes the method according to statement 92, wherein calculating (4010) a local expected garbage collection time (2820) reduces a delay (2825) associated with programmable valid pages within each erase block. The step of adding (4155) is further included.

Statement 94. An embodiment of the present invention includes the method according to statement 92, further comprising: periodically querying the primary replica 2315 for the number of actual empty pages 2805 (4120).

Description 95. An embodiment of the present invention includes the method according to statement 91, wherein calculating (4010) a local expected garbage collection time (2820) includes previous local garbage collection information (3105) for the primary replica (2315). , the local estimated garbage collection time ( 2905), further comprising the step of calculating 4110, 4115.

Statement 96. An embodiment of the present invention includes the method according to statement 83, wherein calculating ( 3920 ) a local estimated time taken ( 3305 ) to complete an I/O request ( 905 ) includes a queue processing time ( 3015 ). The method further includes calculating 4015 .

Description 97. An embodiment of the invention comprises the method according to statement 96, wherein calculating 4015 queue processing time 3015 comprises:

determining (4205, 4220) a queue depth for a queue of pending I/O requests (905) on the primary replica (2315); and

estimating (4210, 4215) the queue processing time 3015 required to process the queue depth.

Description 98. An embodiment of the present invention comprises the method according to statement 97, wherein estimating a queue processing time 3015 required to process a queue depth (4210, 4215) comprises:

determining 4210 a time required to process one I/O request 905 3010; and

and multiplying the time required to process one I/O request (905) (3010) by the queue depth to determine the queue processing time (3015).

Description 99. An embodiment of the present invention comprises the method according to statement 98, wherein determining ( 4210 ) a time required for processing one I/O request ( 905 ) ( 4210 ) comprises: primary replica ( 2315 ) at least one of previous processing time information 3120 on the primary replica 2315, a worst case estimate 3125 for processing time 3125 on the primary replica 2315, and an average case estimate 3130 for processing time 3125 on the primary replica 2315 and determining ( 4210 ) a time required ( 3010 ) to process one I/O request ( 905 ) using

Statement 100. An embodiment of the present invention includes the method according to statement 83, further comprising generating 4025 a local garbage collection weight 2635 and an expected garbage collection weight 2640.

Description 101. An embodiment of the present invention comprises a method according to statement 100, wherein generating 4025 a local garbage collection weight 2635 and an expected garbage collection weight 2640 generates a queue processing weight 2645 Step 4025 is included.

Description 102. An embodiment of the present invention comprises the method according to statement 100, wherein generating 4025 a local garbage collection weight 2635, an expected garbage collection weight 2640, and a queue processing weight 2645 comprises: generating (4410) a local garbage collection weight (2635), an expected garbage collection weight (2640), and a queuing weight (2645) using a linear regression analysis based on previous data for the primary replica (2315) .

Description 103. An embodiment of the invention comprises a method according to statement 102, wherein the old data comes from the use of a sliding window of the primary replica 2315.

Description 104. An embodiment of the present invention comprises a method according to statement 83,

calculating ( 3920 ) an estimated local time taken to complete the I/O request ( 905 ) 3305 further includes calculating ( 4020 ) an estimated local time ( 3205 ); and

Computing 4030 the local estimated time taken 3305 includes a local garbage collection time 2820, a local estimated garbage collection time 2905, an estimated local time 3205, a local garbage collection weight 2635, and an estimated garbage collection time. and calculating 4030 a local estimated time taken 3305 from the collection weights 2640 .

Description 105. An embodiment of the present invention comprises the method according to statement 80, comprising calculating at least one remote estimated time required 3710 for at least one secondary replica 2320, 2325 storing requested data. Step 3940,

calculating (4505) a communication time (3410) for the at least one secondary replica (2320, 2325);

calculating (4510) remote processor time (3515) for at least one secondary replica (2320, 2325);

calculating (4515) a remote garbage collection time (3705) for the at least one secondary replica (2320, 2325); and

at least one remote from communication time 3410 , remote processor time 3515 , remote garbage collection time 3705 , communication time weight 2735 , remote processor time weight 2740 , and remote garbage collection time weight 2745 . and calculating (4540) the estimated required time (3710).

Description 106. An embodiment of the present invention comprises the method according to statement 105, wherein calculating (4530) at least one teleestimated required time (3710) comprises: communicating time weight (2735) to communication time (3410) At least one remote estimate required as the sum of the product, the remote processor time 3515 multiplied by the remote processor time weight 2745, and the remote garbage collection time 3705 times the remote garbage collection time weight 2745 and calculating 4530 time 3710 .

Description 107. An embodiment of the present invention comprises the method according to statement 105, wherein calculating 4505 communication time 3410 for at least one secondary replica 2320, 2325 comprises: ) Checking the communication state of the secondary distributed storage system nodes (125, 130, 135) including the step 4605, at least one secondary replica (2320, 2325) of accessing previous information about the communication time 3410 step 4615, and one of accessing storage graph information for distributed storage system nodes 125, 130, 135 and secondary distributed storage system nodes 125, 130, 135, 4620.

Description 108. An embodiment of the present invention comprises a method according to statement 107, wherein the secondary distributed storage system nodes (125, 130, 135) include secondary replicas (2320, 2325) for determining a communication time (3410). The method further includes periodically inspecting the communication state (4605, 4610).

Description 109. An embodiment of the invention comprises the method according to statement 105, wherein calculating 4510 remote processor time 3515 for at least one secondary replica 2320, 2325 comprises:

querying (4705, 4720) the remote processor for the cost of the remote processor for the at least one secondary replica (2320, 2325); and

and mapping 4735 its cost to remote processor time 3515 .

Statement 110. An embodiment of the present invention comprises the method according to statement 109, wherein the step of inquiring (4705, 4720) of the cost of the remote processor to the remote processor for at least one secondary replica (2320, 2325) comprises at least querying 4705 for a remote processor load 3505 to the remote processor for one secondary replica 2320 , 2325 .

Description 111. An embodiment of the present invention comprises a method according to statement 109, wherein the step of inquiring (4705, 4720) of the cost of the remote processor to the remote processor for at least one secondary replica (2320, 2325) comprises at least and querying (4720) the remote processor for a remote software stack load (3510) for one secondary replica (2320, 2325).

Statement 112. An embodiment of the present invention comprises the method according to statement 109, comprising periodically querying (4705, 4710) the remote processor for a cost for the remote processor for at least one secondary replica (2320, 2325). 4720, 4725).

Description 113. An embodiment of the present invention comprises the method according to statement 105, wherein calculating (4515) a remote garbage collection time (3705) for at least one secondary replica (2320, 2325) comprises:

querying (4120) the actual number of empty pages (2805) to the at least one secondary replica (2320, 2325);

calculating ( 4150 ) a difference value by subtracting the actual number of empty pages ( 2305 ) from a blank page count threshold ( 2810 ) for the at least one secondary replica ( 2320 , 2325 );

and multiplying the difference value by the remote average garbage collection time to determine the remote garbage collection time 3705 .

Description 114. An embodiment of the present invention includes the method according to statement 113, wherein calculating (4515) a remote garbage collection time (3705) for at least one secondary replica (2320, 2325) comprises: each erase block The method further includes adding (4155) a delay (2825) associated with programmable valid pages in .

Description 115. An embodiment of the present invention comprises the method according to statement 113, further comprising periodically querying the at least one secondary replica 2320, 2325 for the actual number of empty pages 2805 (4120, 4125) include

Description 116. An embodiment of the present invention includes the method according to statement 105, wherein calculating (4515) a remote garbage collection time (3705) for at least one secondary replica (2320, 2325) comprises: at least one secondary replica (2320, 2325). Previous remote garbage collection information 3165 for replicas 2320, 2325, worst case estimate 3170 for remote garbage collection 317 on at least one secondary replica 2320, 2325, on at least one secondary replica calculating (4110, 4115) a remote garbage collection time (3705) for at least one secondary replica (2320, 2325) using at least one of an average case estimate (3175) for remote garbage collection. .

Description 117. An embodiment of the present invention comprises a method according to statement 105, comprising generating (4525) a communication time weight (2735), a remote processor time weight (2740), and a remote garbage collection time weight (2745) include more

Description 118. An embodiment of the present invention comprises a method according to statement 117, wherein generating 4525 a communication time weight 2735, a remote processor time weight 2740, and a remote garbage collection time weight 2745 , generating 4410 a communication time weight 2735, a remote processor time weight 2740, and a remote garbage collection time weight 2745 using a linear regression analysis based on previous data of the primary replica 2315. include

Description 119. An embodiment of the invention comprises a method according to statement 118, wherein the old data comes from the use of a sliding window of the primary replica 2315.

Description 120. An embodiment of the invention comprises a method according to statement 105,

Calculating ( 3940 ) at least one remote estimated time required for at least one secondary replica ( 2320 , 2325 ) storing the requested data ( 3940 ) includes calculating ( 4520 ) an estimated remote time ( 3605 ). further comprising;

Computing 4530 at least one remote estimated time required 3710 includes communication time 3410, remote processor time 3515, remote garbage collection time 3705, estimated remote time 3605, communication time weights. calculating 4530 at least one remote estimated time required 3710 from 2735 , the remote processor time weight 2740 , and the remote garbage collection time weight 2745 .

Description 121. An embodiment of the present invention includes an article, including a tangible storage medium, and when executed by a machine, the tangible storage medium comprises:

In the distributed storage system nodes (125, 130, 135), an input/output (I/O) request (905), I requesting data from the primary replica (2315) in the distributed storage system nodes (125, 130, 135) receiving 3905 a primary replica 2315 comprising an /O request 905, storage 140, 145, 150, 155, 160, 165, 225, 230;

calculating ( 3920 ) a local estimated time taken ( 3305 ) to complete an I/O request ( 905 );

calculating ( 3940 ) at least one remote estimated time required ( 3710 ) for at least one secondary replica ( 2320 , 2325 ) storing the requested data;

comparing (3950) the local estimated time taken (3305) to the at least one remote estimated time taken (3710);

selecting (3955) one of the primary replica (2315) and the at least one secondary replica (2320, 2325) in response to the local estimated time spent (3305) and the at least one remote estimated time spent (3710);

It further stores non-transitory instructions that cause the I/O request 905 to be sent to a selected one of a primary replica 2315 and at least one secondary replica 2320 , 2325 .

Description 122. An embodiment of the present invention comprises the object according to statement 121, wherein the step 3905 of receiving the I/O request 905 at the distributed storage system node 125, 130, 135 comprises: I/O request 905 at (125, 130, 135), I/O request 905 requesting data from primary replica 2315 at distributed storage system nodes 125, 130, 135, solid state drive and receiving 3905 a primary replica 2315 comprising (SSD) 140 , 145 , 150 , 155 , 160 , 165 , 225 , 230 .

Description 123. An embodiment of the present invention comprises an article according to statement 121, wherein the distributed storage system nodes 125, 130, 135 emerge from a set comprising a network-attached solid state drive (SSD) and an Ethernet SSD.

Description 124. An embodiment of the present invention comprises an article according to statement 121, and an embodiment of the present invention comprises an article, comprising a tangible storage medium, and when executed by a machine, the tangible storage medium comprises: It further stores non-transitory instructions that result in a step 3910 of performing this method only while the primary replica 2315 is performing garbage collection.

Description 125. An embodiment of the present invention includes an article according to statement 124, wherein an embodiment of the present invention includes an article, comprising a tangible storage medium, wherein when executed by a machine, the tangible storage medium comprises:

comparing 3925 the local estimated time taken 3305 to a threshold time 2525; and

If the local estimated turnaround time 3305 is less than the threshold time 2525 , it stores more non-transitory instructions that result in a step 3915 processing the I/O request 905 at the primary replica 2315 .

Statement 126. An embodiment of the present invention comprises the product according to statement 125, wherein the processing 3915 of the I/O request 905 at the primary replica 2315 comprises at least one of at least one storing the requested data. calculating ( 3940 ) at least one remote estimated time spent ( 3710 ) for the secondary replicas ( 2320 , 2325 ), and comparing the local estimated time spent ( 3305 ) to the at least one remote estimated time spent ( 3710 ). and processing 3915 the I/O request 905 at the primary replica 2315 without 3950 .

Description 127. An embodiment of the present invention comprises the object according to statement 124, wherein calculating 3920 a local estimated time taken 3305 to complete an I/O request 905 comprises:

calculating (4005) a local garbage collection time (2820);

calculating 4010 a local expected garbage collection time 2905;

calculating 4030 a local estimated time taken 3305 from a local garbage collection time 2820, a local budget garbage collection time 2905, a local garbage collection weight 2635, and an expected garbage collection weight 2640 do.

Description 128. An embodiment of the present invention includes the object according to description 127, wherein the step 4030 of calculating the local estimated time taken 3305 includes the local garbage collection weight 2635 for the local garbage collection time 2820. and calculating 4030 the local estimated required time 3305 as the sum of the multiplied value and the local estimated garbage collection time 2905 multiplied by the expected garbage collection weight 2640 .

Description 129. An embodiment of the present invention includes the object according to description 128, and the step 4030 of calculating a local estimated time taken 3305 includes: a local garbage collection weight 2635 for a local garbage collection time 2820. Local estimated time taken 3305 as the sum of the product multiplied by the local estimated garbage collection time 2905 times the expected garbage collection weight 2640, and the queue processing time 3015 multiplied by the queuing weight 2645. It further comprises the step of calculating

Description 130. An embodiment of the present invention comprises the object according to statement 127, wherein calculating a local garbage collection time 2820 (4005) comprises:

determining (4105) whether the primary replica 2315 is performing garbage collection; and

and calculating 4005 a local garbage collection time 2820 if the primary replica 2315 is currently performing garbage collection.

Description 131. An embodiment of the present invention includes the object according to description 130, and calculating a local garbage collection time 2820 comprises:

querying the primary replica 2315 for the actual number of empty pages 2805;

calculating ( 4105 ) a difference value of the number of empty pages threshold ( 2810 ) for the primary replica ( 2315 ) minus the actual number of empty pages ( 2805 );

The method further includes multiplying the difference value by a local average garbage collection time 2815 to determine a local garbage collection time 2820 .

Description 132. An embodiment of the present invention includes the object according to statement 131, wherein calculating a local garbage collection time 2820 (4005) adds a delay 2820 associated with programmable valid pages within each erase block. Step 4155 is further included.

Description 133. An embodiment of the present invention includes an object according to statement 131, wherein an embodiment of the present invention includes an object, including a tangible storage medium, and when executed by a machine, the tangible storage medium comprises: It further stores non-transitory instructions that cause the primary replica 2315 to periodically query the actual number of empty pages 2805 ( 4120 , 4125 ).

Description 134. An embodiment of the present invention comprises the object according to statement 130, old local garbage collection information 3105 on primary replica 2315, worst case of local garbage collection 3110 on primary replica 2315 The method further includes calculating (4110, 4115) a local garbage collection time (2820) using at least one of a case estimate, an average case estimate for a local garbage collection (3115) on the primary replica (2315).

Description 135. An embodiment of the present invention comprises the object according to statement 127, wherein calculating 4010 a local expected garbage collection time 2905 comprises:

determining whether the primary replica 2315 is expected to begin garbage collection soon; and

and calculating 4010 an expected garbage collection time 2820 only if the primary replica is about to perform garbage collection soon.

Description 136. An embodiment of the present invention comprises the object according to statement 135, wherein calculating 4010 a local expected garbage collection time 2820 comprises:

querying 4120 the number of actual blank pages 2820 to the primary replica 2315;

calculating ( 4150 ) a difference value of the number of empty pages threshold ( 2810 ) for the primary replica ( 2315 ) minus the actual number of empty pages ( 2805 ); and

The method further comprises a step 4160 of multiplying the difference by a local average garbage collection time 2815 to determine a local estimated time required 3305 .

Description 137. An embodiment of the present invention includes the object according to statement 136, wherein calculating a local expected garbage collection time 2820 4010 reduces a delay 2825 associated with programmable valid pages within each erase block. The step of adding (4155) is further included.

Description 138. An embodiment of the present invention comprises an article according to statement 136, wherein an embodiment of the present invention comprises an article, comprising a tangible storage medium, wherein when executed by a machine, the tangible storage medium comprises: It further stores non-transitory instructions that result in a step 4120 of periodically querying the primary replica 2315 for the actual number of empty pages 2805.

Description 139. An embodiment of the present invention comprises the object according to statement 135, wherein the step 4010 of calculating a local expected garbage collection time 2820 comprises previous local garbage collection information 3105 for the primary replica 2315 , the local estimated garbage collection time ( 2905), further comprising the step of calculating 4110, 4115.

Description 140. An embodiment of the present invention comprises an object according to statement 127, wherein calculating 3920 a local estimated time taken 3305 to complete an I/O request 905 comprises a queue processing time 3015 The method further includes calculating 4015 .

Description 141. An embodiment of the present invention includes the object according to description 140, and the step 4015 of calculating the queue processing time 3015 includes:

determining (4205, 4220) a queue depth for a queue of pending I/O requests (905) on the primary replica (2315); and

estimating (4210, 4215) the queue processing time 3015 required to process the queue depth.

Description 142. An embodiment of the present invention includes the object according to description 141, and the steps of estimating the queue processing time 3015 required to process the queue depth (4210, 4215) include:

determining 4210 a time required to process one I/O request 905 3010;

and multiplying (4215) the time required to process one I/O request (905) 3010 by the queue depth to determine the queue processing time (3015).

Description 143. An embodiment of the present invention comprises an object according to statement 142, wherein the step of determining ( 4210 ) a time required to process one I/O request ( 905 ) 3010 comprises: a primary replica ( 2315 ) at least one of previous processing time information 3120 on the primary replica 2315, a worst case estimate 3125 for processing time 3125 on the primary replica 2315, and an average case estimate 3130 for processing time 3125 on the primary replica 2315 and determining ( 4210 ) a time required to process one I/O request ( 905 ) 3010 .

Description 144. An embodiment of the present invention comprises an article according to statement 127, wherein an embodiment of the present invention comprises an article, comprising a tangible storage medium, wherein when executed by a machine, the tangible storage medium comprises: It further stores non-transient instructions that result in a step 4025 of generating a local garbage collection weight 2635 and an expected garbage collection weight 2640 .

Description 145. An embodiment of the present invention includes the object according to description 144, wherein the step 4025 of generating a local garbage collection weight 2635 and an expected garbage collection weight 2640 includes generating a queue processing weight 2645 Step 4025 is included.

Description 146. An embodiment of the present invention includes the object according to description 144, wherein generating 4025 a local garbage collection weight 2635, an expected garbage collection weight 2640, and a queue processing weight 2645 comprises: generating (4410) local garbage collection weights (2635), expected garbage collection weights (2640), and queuing weights (2645) using linear regression analysis based on previous data for primary replica (2315) .

Description 147. An embodiment of the present invention comprises an object according to statement 144, wherein the old data comes from the use of a sliding window of the primary replica 2315.

Description 148. An embodiment of the present invention comprises an article according to statement 127,

calculating ( 3920 ) an estimated local time taken to complete the I/O request ( 905 ) 3305 further includes calculating ( 4020 ) an estimated local time ( 3205 ); and

Computing 4030 the local estimated time taken 3305 includes a local garbage collection time 2820, a local estimated garbage collection time 2905, an estimated local time 3205, a local garbage collection weight 2635, and an estimated garbage collection time. and calculating 4030 a local estimated time taken 3305 from the collection weights 2640 .

Description 149. An embodiment of the invention comprises the object according to statement 124, comprising calculating at least one remote estimated time required 3710 for at least one secondary replica 2320, 2325 storing the requested data. Step 3940,

calculating (4505) a communication time (3410) for the at least one secondary replica (2320, 2325);

calculating (4510) remote processor time (3515) for at least one secondary replica (2320, 2325);

calculating (4515) a remote garbage collection time (3705) for the at least one secondary replica (2320, 2325); and

at least one remote from communication time 3410 , remote processor time 3515 , remote garbage collection time 3705 , communication time weight 2735 , remote processor time weight 2740 , and remote garbage collection time weight 2745 . and calculating (4540) the estimated required time (3710).

Description 150. An embodiment of the present invention includes the object according to statement 149, wherein the step 4530 of calculating at least one remote estimated required time 3710 comprises: a communication time weight 2735 equal to a communication time 3410 At least one remote estimate required as the sum of the product multiplied by the remote processor time 3515 times the remote processor time weights 2745 and the remote garbage collection time 3705 times the remote garbage collection time weights 2745 and calculating 4530 time 3710 .

Description 151. An embodiment of the present invention comprises an object according to statement 149, wherein calculating 4505 communication time 3410 for at least one secondary replica 2320, 2325 comprises: ) Checking the communication state of the secondary distributed storage system nodes (125, 130, 135) including the step 4605, at least one secondary replica (2320, 2325) of accessing previous information about the communication time 3410 step 4615, and one of accessing storage graph information for distributed storage system nodes 125, 130, 135 and secondary distributed storage system nodes 125, 130, 135, 4620.

Description 152. An embodiment of the present invention comprises an article according to statement 151, an embodiment of the present invention comprises an article, comprising a tangible storage medium, and when executed by a machine, the tangible storage medium comprises: A rain that causes a step 4605, 4610 of periodically checking the communication state of the secondary distributed storage system nodes 125, 130, 135, including the secondary replicas 2320, 2325, to determine the communication time 3410 Store more temporary commands.

Description 153. An embodiment of the invention comprises the object according to statement 149, wherein calculating 4510 remote processor time 3515 for at least one secondary replica 2320, 2325 comprises:

querying (4705, 4720) the remote processor for the cost for the remote processor for the at least one secondary replica (2320, 2325); and

and mapping 4735 its cost to remote processor time 3515 .

Description 154. An embodiment of the invention comprises the product according to statement 153, wherein the step of inquiring (4705, 4720) of the cost of the remote processor to the remote processor for at least one secondary replica (2320, 2325) comprises at least and querying (4705) the remote processor for a remote processor load (3505) for one secondary replica (2320, 2325).

Description 155. An embodiment of the present invention comprises an article according to statement 153, wherein the step of inquiring (4705, 4720) of the cost of the remote processor to the remote processor for at least one secondary replica (2320, 2325) comprises at least and querying (4720) the remote processor for a remote software stack load (3510) for one secondary replica (2320, 2325).

Description 156. An embodiment of the present invention includes an article according to statement 153, wherein an embodiment of the present invention includes an article, comprising a tangible storage medium, wherein when executed by a machine, the tangible storage medium comprises: It further stores non-transitory instructions that cause the remote processor for the at least one secondary replica 2320 , 2325 to periodically query the cost for the remote processor 4705 , 4710 , 4720 , 4725 .

Description 157. An embodiment of the present invention comprises the object according to statement 149, wherein calculating (4515) a remote garbage collection time (3705) for at least one secondary replica (2320, 2325) comprises:

querying (4120) the actual number of empty pages (2805) to the at least one secondary replica (2320, 2325);

calculating ( 4150 ) a difference value by subtracting the actual number of empty pages ( 2305 ) from a blank page count threshold ( 2810 ) for the at least one secondary replica ( 2320 , 2325 );

and multiplying the difference value by the remote average garbage collection time to determine the remote garbage collection time 3705 .

Description 158. An embodiment of the present invention comprises the object according to statement 157, wherein calculating 4515 a remote garbage collection time 3705 for at least one secondary replica 2320, 2325 comprises: each erase block The method further includes adding (4155) a delay (2825) associated with programmable valid pages in .

Description 159. An embodiment of the present invention comprises an article according to statement 157, wherein an embodiment of the present invention comprises an article, comprising a tangible storage medium, wherein when executed by a machine, the tangible storage medium comprises: It further stores non-transitory instructions that cause a step 4120 , 4125 to periodically query the at least one secondary replica 2320 , 2325 for the actual number of empty pages 2805 .

Description 160. An embodiment of the present invention comprises the object according to statement 157, wherein calculating 4515 a remote garbage collection time 3705 for at least one auxiliary replica 2320, 2325 comprises: Previous remote garbage collection information 3165 for replicas 2320, 2325, worst case estimate 3170 for remote garbage collection 317 on at least one secondary replica 2320, 2325, at least one secondary replica calculating (4110) a remote garbage collection time (3705) for at least one secondary replica (2320, 2325) using at least one of an average case estimate (3175) for remote garbage collection on 2320, 2325; 4115).

Description 161. An embodiment of the present invention includes an object according to statement 149, wherein an embodiment of the present invention includes an object, including a tangible storage medium, wherein when executed by a machine, the tangible storage medium comprises: It further stores non-transitory instructions that result in generating 4525 communication time weight 2735 , remote processor time weight 2740 , and remote garbage collection time weight 2745 .

Description 162. An embodiment of the present invention comprises the object according to statement 161, wherein the step 4525 of generating a communication time weight 2735, a remote processor time weight 2740, and a remote garbage collection time weight 2745 comprises: , generating 4410 a communication time weight 2735, a remote processor time weight 2740, and a remote garbage collection time weight 2745 using a linear regression analysis based on previous data of the primary replica 2315. include

Description 163. An embodiment of the invention comprises an object according to statement 162, wherein the old data comes from the use of a sliding window of the primary replica 2315.

Description 164. An embodiment of the invention comprises an article according to statement 149,

Computing ( 3940 ) at least one remote estimated time required for at least one secondary replica ( 2320 , 2325 ) storing the requested data comprises ( 3940 ) calculating ( 4520 ) an estimated remote time ( 3605 ). further comprising;

Computing 4530 at least one remote estimated time required 3710 includes a communication time 3410 , a remote processor time 3515 , a remote garbage collection time 3705 , an estimated remote time 3605 , and a communication time weight. calculating 4530 at least one remote estimated time required 3710 from 2735 , the remote processor time weight 2740 , and the remote garbage collection time weight 2745 .

In conclusion, in view of various modifications to the embodiments described herein, this detailed description and accompanying material are intended to be illustrative only, and should not be construed as limiting the scope of the present invention. Therefore, what is claimed as the present invention are all modifications that may come within the scope and spirit of the following claims and their equivalents.

Claims (10)

at least one storage device containing a primary copy of data;
A local estimated time required to complete an input/output (I/O) request using the primary replica and at least one remote estimated time required to complete the I/O request using at least one secondary replica of the data. an analyzer that counts time; and
a relocator for forwarding the I/O request to one of the primary replica and the at least one secondary replica based on the local estimated duration and the at least one remote estimated duration;
The local estimated time required includes an estimated time required to process a pending I/O request,
the analyzer includes a local time estimator for calculating the local estimated required time;
The local time estimator is:
local time calculator to calculate local garbage collection time;
Local Estimated Time Calculator for calculating local Estimated Garbage Collection Time;
a second storage for storing local garbage collection weights and expected garbage collection weights; and
A distributed storage system node comprising: a local estimated time calculator for calculating the local estimated required time based on at least a part of the local garbage collection time, the local expected garbage collection time, the local garbage collection weight, or the expected garbage collection weight . The method of claim 1,
and the relocator is operative to forward the I/O request based on at least a portion of the at least one storage device performing garbage collection. 3. The method of claim 2,
The relocation is:
a first storage for storing a threshold time; and
and a comparator for comparing the local estimated required time and the threshold time. delete The method of claim 1,
The analyzer is:
query logic to query the primary replica for the number of pages in an available state; and
receiving logic to receive the number of pages in the usable state from the primary replica;
The local time calculator calculates a difference value by subtracting the number of pages in the usable state from the threshold value of pages in the usable state of the primary replica, and multiplies the difference value by the local average garbage collection time to determine the local garbage collection time A distributed storage system node that computes 6. The method of claim 5,
and the local time calculator is operative to add a delay associated with programming valid pages in an erase block to the local garbage collection time. The method of claim 1,
and the relocator is operative to forward the I/O request to the primary replica based on at least a portion of the at least one remote estimated duration exceeding the local estimated duration. The method of claim 1,
and the relocator is operative to forward the I/O request to the at least one secondary replica based on at least a portion of the local estimated duration exceeding the at least one remote estimated duration. receiving, at a distributed storage system node, an input/output (I/O) request requesting data from a primary replica comprising a storage device;
calculating a local estimated time required to complete the I/O request;
calculating at least one remote estimated time required for at least one secondary replica storing the requested data;
comparing the at least one remote estimated required time with the local estimated required time;
selecting one of the primary replica and the at least one secondary replica based on the shortest of the local estimated time required and the at least one remote estimated required time; and
forwarding the I/O request to the selected one of the primary replica and the at least one secondary replica;
The local estimated time required includes an estimated time required to process a pending I/O request,
Calculating the estimated local time required to complete the I/O request comprises:
calculating a local garbage collection time;
calculating a local estimated garbage collection time;
and calculating the local estimated required time based on at least a part of the local garbage collection time, the local expected garbage collection time, a local garbage collection weight, or an expected garbage collection weight.
delete

Images