KR20230114178A - Low latency multiple storage device system - Google Patents

Abstract

system is initiated. A storage device may store data. A load module may read the data from the storage device based at least in part on input/output (I/O). A scheduler may place the I/O request into a queue for delivery to the load module based at least in part on a size of the I/O request that is less than a threshold.

Description

Multiple storage device system with low latency {LOW LATENCY MULTIPLE STORAGE DEVICE SYSTEM}

The present disclosure relates generally to storage devices, and more particularly to systems including storage devices enabling low read latency.

Some applications may depend on large amounts of data. If this data is not yet stored in memory, it can be read from storage devices. However, accessing data from a single storage device can create a bottleneck that slows down data access and subsequent operations.

The need to provide low latency access to data remains.

One object of the present disclosure is to provide systems and methods that achieve high query per second (QPS) and low latency.

Embodiments of the present disclosure may include a system. The system may include a storage device for storing data and a load module for reading the data from the storage device. The scheduler may receive an input/output (I/O) request and transfer the input/output request to the load module based on the size of the input/output request.

A system according to embodiments of the present disclosure may include an operation scheduler. The operation scheduler may place the operation request into one of the queues based on the workload of the operation request. The operation scheduler may select a processing element and transmit the operation request to the selected processing element. The operation scheduler may operate based on operation intensity.

A system according to embodiments of the present disclosure includes a multi-processing system. The multiprocessing system may load data from an I/O processing/storage pool using I/O requests, which may be based on data being processed using operation requests. Data may be retrieved using load modules associated with the storage devices of the I/O processing/storage pool. With the above configuration, the system can achieve high QPS and low latency.

The drawings described below are examples of how embodiments of the present disclosure may be implemented, and are not intended to limit the embodiments of the present disclosure. Individual embodiments of the present disclosure may include elements not shown in certain figures and/or may omit elements shown in certain figures. The drawings are for illustrative purposes only and may not be to scale.
1 illustrates an apparatus configured to support low latency access to storage devices in processing operations according to embodiments of the present disclosure.
Figure 2 shows details of the apparatus of Figure 1 according to embodiments of the present disclosure.
FIG. 3 illustrates details of the multiple processing system of FIG. 1 according to embodiments of the present disclosure.
4 illustrates details of an I/O request issued to the multi-processing system of FIG. 1 according to embodiments of the present disclosure.
Figure 5 shows details of the table of Figure 3 according to embodiments of the present disclosure.
6 illustrates details of the I/O scheduler of FIG. 3 according to embodiments of the present disclosure.
7 shows details of the load module of FIG. 3 according to embodiments of the present disclosure.
8 illustrates details of the computing system of FIG. 1 according to embodiments of the present disclosure.
FIG. 9 illustrates a flowchart of an exemplary procedure for processing the I/O request of FIG. 4 using the multiprocessing system of FIG. 1 in accordance with embodiments of the present disclosure.
FIG. 10 depicts an alternative flow diagram of an exemplary procedure for processing the I/O request of FIG. 4 using the multiprocessing system of FIG. 1 in accordance with embodiments of the present disclosure.
11 illustrates a flowchart of an exemplary procedure for the I/O scheduler of FIG. 3 to use priority queuing for queuing the I/O request of FIG. 4 in accordance with embodiments of the present disclosure.
FIG. 12 illustrates a flowchart of an exemplary procedure for the manager of FIG. 3 to allocate the I/O request of FIG. 4 to the load module of FIG. 3 according to embodiments of the present disclosure.
13 illustrates a flowchart of an exemplary procedure for the load module of FIG. 3 to read data from the storage device of FIG. 1 according to embodiments of the present disclosure.
14 depicts a flowchart of an exemplary procedure for processing the computational request of FIG. 8 by the computational system of FIG. 1 in accordance with embodiments of the present disclosure.
15A and 15B show a flow diagram of an exemplary procedure in which the operation scheduler of FIG. 8 arranges the processing elements of FIG. 8 to process the operation request of FIG. 8, in accordance with embodiments of the present disclosure.

Reference will now be made in detail to embodiments of the present disclosure, which examples are illustrated in the accompanying drawings. In the detailed description that follows, numerous specific details are set forth for a thorough understanding of the present disclosure. However, it should be understood that one of ordinary skill in the art may practice the present disclosure without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments of the present disclosure.

Although the terms first, second, etc. may be used herein to describe various elements, it will be understood that these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first module could be termed a second module, and similarly, a second module could be termed a first module, without departing from the scope of the present disclosure.

Terms used in the description of the present disclosure are only used to describe specific embodiments and are not intended to limit the present disclosure. As used in the description of this disclosure and the appended claims, the singular forms are intended to include the plural forms unless the context clearly dictates otherwise. Also used herein, the term "and/or" can be understood to refer to and include any and all possible combinations of one or more of the related listed items. The terms “comprise” and/or “comprising”, when used in this disclosure, specify the presence of specified features, integers, steps, operations, elements, and/or components but not one or more other features. does not exclude the presence or addition of s, integers, steps, operations, elements, components, and/or groups thereof. Components and features of the drawings are not necessarily drawn to scale.

Some applications, such as deep learning recommendation models (DLRMs), can rely on large amounts of data. DLRMs can rely on built-in tables that can be terabytes in size. Transferring large amounts of data from storage to memory for processing can take time. If multiple applications attempt to process the data stored on the storage device, this problem can be exacerbated as the storage device may have insufficient bandwidth limitations for all the requested data.

Embodiments of the present disclosure address these problems by introducing a system of storage devices. The storage devices may be, for example, Solid State Drives (SSDs). Data can be distributed across storage devices to provide all requested data by reducing the load on individual storage devices. A scheduler can schedule I/O requests into one or more queues based on the size of the data to be retrieved. The I/O processing manager can retrieve requests from queues and can identify load modules to retrieve data from storage devices.

1 illustrates an apparatus configured to support low latency access to storage devices in processing operations according to embodiments of the present disclosure. 1, device 105, which may also be referred to as a host or system, includes a processor 110, memory 115, and storage devices 120-1 and 120-2 (collectively referred to as storage device 120). can be). Processor 110 may be any of a variety of processors (processor 110, along with other components discussed below, is shown external to the device for convenience of explanation. Embodiments of the present disclosure provide such components may be included). 1 depicts a single processor 110, device 105 may include any number of processors, each of which may be single-core or multi-core processors, each of which may be (other than Among possible architectures, it may implement a Reduced Instruction Set Computer (RISC) architecture or a Complex Instruction Set Computer (CISC) architecture, and may be mixed in any desired combination.

Processor 110 may be coupled with memory 115 . The memory 115 may be any of a variety of memories such as flash memory, dynamic random access memory (DRAM), static random access memory (SRAM), persistent random access memory (PRAM), ferroelectric random access memory (FRAM), or magnetoresistive memory (MRAM). It may be a Non-Volatile Random Access Memory (NVRAM) such as Random Access Memory). Memory 115 may be volatile or non-volatile memory as desired. Memory 115 may also be any desired combination of different memory types and may be managed by memory controller 125 . Memory 115 may be used to store data that may be referred to as “short-term,” i.e., data that is not expected to be stored for long periods of time. Examples of short-lived data may include temporary files, data that may be used locally by applications (which may have been copied from other storage locations), and the like.

Processor 110 and memory 115 may also support an operating system on which various applications may be executed. These applications may issue requests (which may also be referred to as commands) to read data from or write data to memory 115 . When storage device 120 is used to support applications that read or write data through some kind of file system, storage device 120 may be accessed using device driver 130 . 1 shows two storage devices 120, there may be any number (one or more) of storage devices in device 105. Storage device 120 may each support any desired protocol or protocols including, for example, the Non-Volatile Memory Express (NVMe) protocol. Different storage devices 120 may support different protocols and/or interfaces.

1 uses the generic term “storage”, embodiments of this disclosure may include any storage device types that can benefit from the use of computational storage units, such as hard disk drives and SSDs. (Solid State Drives). All references to “SSD” below should be understood to include such other embodiments of the present disclosure. Also, different types of storage devices may be mixed. For example, the storage device 120-1 may be a hard disk drive, and the storage device 120-2 may be an SSD.

Device 105 may also include multiple processing system 135 and computing system 140 . Multi-processing system 135 is configured to process storage devices 120 based on input/output (I/O) requests received from applications running on processor 110 (or processors of remote devices not shown in FIG. 1). You can manage reading data from . The I/O requests may request data from storage devices 120 that may be used in computing processes. That is, when an operation request is given, data to be processed by the operation request may be requested from the storage device 120 in an input/output request that is first processed by the multiprocessing system 135 . Multiprocessing system 135 bases its storage on the size of the I/O request to reduce latency (the time required to complete the computational processes, including the time required to read data and execute appropriate instructions on the data). Reading data from devices 120 can be scheduled. Multiple processing system 135 is discussed further with reference to FIG. 3 below.

Once data is read by multiprocessing system 135, computational system 140 may execute computational processes to process the data. Computing system 140 is discussed further with reference to FIG. 8 below.

As noted above, device 105 may include multiple storage devices 120 . By including one or more storage devices 120 , data requested in an I/O request may be distributed across the storage devices 120 . By distributing the data across storage devices 120, read requests can be processed by each storage device 120. When these read requests are processed in parallel, the requested data can be read faster than when all data is stored in only one storage device 120 . However, embodiments of the present disclosure may include one storage device 120 (without the potential benefit of reading data from multiple storage devices 120 in parallel).

1 shows device 105 as including multiple processing system 135 and computing system 140, embodiments of the present disclosure may have these components located elsewhere. For example, multiprocessing system 135 may be included as part of device 105 while computing system 140 may be part of another device reaching over a network. Indeed, embodiments of the present disclosure may separate storage devices 120, multiple processing system 135, and computing system 140, respectively, into separate devices 105 that are connected via a network or communication path.

2 shows details of the apparatus of FIG. 1 according to embodiments of the present disclosure. 2, device 105 generally includes one or more processors 110, which control a memory controller 120 and clocks 205 that can be used to coordinate the operations of components of device 105. can include Processors 110 may also be coupled with memories 115 , which may include, for example, random access memory (RAM), read-only memory (ROM), or other state-preserving media. Processors 110 may also be coupled to storage devices 120 and may be coupled to network connector 210, which may be, for example, an Ethernet connector or a wireless connector. Processors 110 may also be coupled to buses 215, which may include user interfaces 220 and input/output that may be managed using input/output (I/O) engines, among other components. (I/O) interface ports may be attached.

Before describing the structure and operation of the multi-processing system 135 of FIG. 1, it may be helpful to consider the various requests that may be processed using the device 105 of FIG. An example application that may run on processor 110 of FIG. 1 is a deep learning recommendation model (DLRM), which is one example of a machine learning algorithm. The DLRM application may have set a service level agreement (SLA) that can govern (determine) the time required to process a query. That is, the DLRM can predict that a specific query will take a certain amount of time. If the query takes longer than that, the DLRM may wait longer than expected before continuing processing.

To execute the query, the device 105 of FIG. 1 may have to retrieve the data in question and then perform computations on that data. Retrieving the data and processing the operations may take some time.

If the query is relatively small (eg, contains less than 256 data points), retrieving data can be relatively fast, and the entire process of executing the query can meet the SLA. However, if the query is relatively large (eg, contains more than 256 data points), retrieving the data may take so long that it does not meet the SLA. Since the DLRM can have a variety of query sizes, relatively large queries can be expected to occur, and the process of retrieving stored data can take longer than the time required to execute the computational process. Since it is undesirable for the device 105 of FIG. 1 to fail to meet the SLA, reducing the time required to retrieve data from the storage devices 120 of FIG. 1 ie achieving low latency for data retrieval it is desirable

FIG. 3 shows details of the multiple processing system of FIG. 1 in accordance with embodiments of the present disclosure. In FIG. 3 , multiple processing system 135 includes an input/output (I/O) scheduler 305, queues 310-1, 310-2, and 310-3 (which may be referred to collectively as queues 310), and an input/output (I/O) process/storage pool 315 .

The I/O scheduler 305 may receive I/O requests from applications running on the processor 110 of FIG. 1 . These requests may identify data to be read from storage devices 120 for use in the computational request (to be processed by computational system 140 of FIG. 1).

The I/O scheduler 305 may determine the size of the I/O requests, that is, the amount of data read from the storage devices 120 used to execute the calculation process. Using the size of the I/O request, the I/O scheduler 305 may select one of the queues 310 to place the I/O request.

In Figure 3, three cues 310-1 to 310-3 are shown. Each queue 310 may be used to store I/O requests of various sizes. For example, queue 310-1 may be used to store I/O requests for retrieving data (rows of an embedding table) of size 128 vectors or less. Queue 310-2 may be used to store I/O requests to retrieve data greater than 128 vectors in size but not greater than 512 vectors. Queue 310-3 may be used to store I/O requests for retrieving data whose size is greater than 512 vectors. In this way, I/O requests can be grouped based on the approximate amount of data to retrieve, which can indicate the time required to retrieve the data.

Although FIG. 3 shows three cues 310 , embodiments of the present disclosure may include any number (one or more) of cues 310 . For example, when priority queuing is used, the I/O scheduler 305 may determine the priority of the I/O request based on the size of data to be read from the storage devices 120, , a priority tag may be associated with an I/O request in queue 310 so that I/O processing/storage pool 315 can determine the relative priority of the I/O request. For example, if the I/O request requests data having a size of 128 vectors or less (embedding vectors (rows of an embedding table) with a size of 128 vectors or less), the priority tag indicates that the request takes precedence. It may indicate having a rank of '1'. When the I/O request requests data having a size greater than 128 vectors but not greater than 512 vectors, the priority tag may indicate that the request has a priority of '2'. When the I/O request requests data having a size greater than 512 vectors, the priority tag may indicate that the request has a priority of '3'. As with the number of queues 310, any number of different priorities may be used. The number of priorities discussed above is merely illustrative.

In some embodiments of the present disclosure, queues 310 may be first in, first out (FIFO) queues. In other embodiments of the present disclosure, other types of queues 310 may be used.

Even if there is only one queue 310, the selection of the queue may still technically be based on the size of the I/O request (although all I/O requests may be placed in that queue). Also, if there is only one queue, the queue may not be a FIFO queue. That is, the I/O requests may be removed from the queue in an order different from the order in which they were added to the queue (e.g., I/O requests with priority '1' added to the queue later than I/O requests with priority '2'). request may still be removed from the queue first and processed first).

When I/O scheduler 305 places an I/O request in queue 310, I/O processing/storage pool 315 may retrieve the I/O request from queues 310. By using multiple queues 310 (or using different priorities), I/O processing/storage pool 315 can select the next I/O request to process. In this way, I/O requests may be processed by I/O processing/storage pool 315 in a different order than the order in which they are sent from applications running on processor 110 of FIG. 1 to multiprocessing system 135.

I/O processing/storage pool 315 may retrieve I/O requests from queues 315 using any desired technique. For example, I/O processing/storage pool 315 uses round-robin access to retrieve I/O requests from queue 310-1, then retrieves I/O requests from queue 310-2, then queues After retrieving the I/O request from 310-3, it can be retrieved by going back to the queue 310-1, etc. (of course, if the queue 310 does not have an I/O request, the I/O processing/storage pool ( 315) skips the corresponding queue 315 and moves to the next queue 315 to retrieve an input/output request.)

I/O processing/storage pool 315 may include a manager 320 that may be responsible for retrieving I/O requests from queues 315 . Manager 320 may also determine which storage device(s) 120 is the requested data (the data may be stored on a single storage device 120, or may be stored on multiple storage devices 120). ) may be responsible for determining whether the When manager 320 determines which storage device(s) 120 is storing the requested data, manager 320 sends the I/O request to load module(s) 325-1, 325-2, 325-3, 325-4 (which may be referred to collectively as load module(s) 325) to read data from storage device(s) 120.

In some embodiments of the present disclosure, each storage device 120 may be considered distinct from other storage devices 120 . That is, there may be no predetermined relationship between the storage devices 120 that manages the use of the storage device 120 . For example, each storage device 120 is not only a physically separated storage device, but also a logically separated storage device (this method, for example, data storage management is RAID (Redundant Array of Independent Disks)) It can be considered as a RAID (comparable to RAID), which is left to the controller. However, in other embodiments of the present disclosure, storage devices 120 may be configured in an array such as a RAID.

To determine which storage device(s) 120 is storing the data requested in the I/O request, manager 320 may access table 330. The table 330 may function similarly to a flash translation layer (FTL) of an SSD. However, instead of mapping a logical address (such as the address of a logical block of data used by an application) to a physical address (of a storage device), table 330 maps the logical address (or some other identifier of data) to the request. It may be mapped to the identifier of the storage device(s) 120 that stores the data. Table 330 may be stored in some form of storage (eg, volatile storage such as local DRAM or non-volatile storage such as firmware module or flash storage). The use of table 330 is described further below with reference to FIG. 5 below.

When manager 320 dispatches the I/O request to load module(s) 325, load module(s) 325 can access the requested data from storage device(s) 120. For example, if data is stored in storage device 120-1, manager 320 may dispatch the I/O request to load modules 325-1 and/or 325-2. If the data is stored in the storage device 120-2, the manager 320 may dispatch the I/O request to the load modules 325-3 and/or 325-4. In FIG. 3 , input/output processing/storage pool 315 is shown as including two storage devices 120 and four load modules 325 , embodiments of the present disclosure may include any number (one or more) of storage devices 120 and any number (one or more) of load modules 325 (although at least one load module 325 must exist for each storage device 120). ).

Note in FIG. 3 that each storage device 120 has two load modules 325 that can access data from the storage device. In some embodiments of the present disclosure, storage devices 120 may support multi-threaded access. That is, storage devices 120 may support reading data to simultaneously satisfy multiple requests. For example, if storage device 120 includes multiple channels, each of which can be used independently of the other channels, one thread can request stored data along a first channel and another thread can request data stored along a second channel. You can request stored data along a channel, and the two threads can run concurrently. In embodiments of the present disclosure that include multiple load modules 325 that can access storage device 120 concurrently, but data can be accessed by only one load module 325, table 330 may also include the identifier of the particular load module 325 used to retrieve the requested data. And in some embodiments of the present disclosure, there may be only one load module 325 per storage device 120 . Embodiments of the present disclosure may also include combinations of these possibilities. For example, one storage device 120 may support multiple threads and may be accessed by multiple load modules 325, while another storage device 120 does not support multiple threads and thus only It can only be accessed by one load module 325.

In FIG. 3 , the load modules 325 may be sparse length sum (SLS) load modules 325 . SLS load modules 325 are discussed further with reference to FIG. 7 below.

The load modules 325 may use, for example, a user-space non-volatile memory express (UNVMe) driver to access the storage devices 120 . While drivers may use a file system by applications to access storage devices 120 , UNVMe drivers may access data directly from storage devices 120 and may not use a file system. Load modules 325 may also use various application programming interfaces (APIs) provided by storage devices 120 to access data.

As described above with reference to Figure 1, storage devices 120 may be any desired variety of storage devices, such as hard disk drives and SSDs. Also, variations of these various types may also be used. For example, storage devices 120 may be SSDs optimized to store and retrieve data to satisfy DLRM queries. Such SSDs may have a different architecture than SSDs intended for general use.

In some embodiments of the present disclosure, data requested in an input/output request may be stored in a specific storage device 120 . As a result, multiple I/O requests can be sent to the same storage device 120 . When accessing data from a particular storage device 120, load modules 325 can manage multiple requests for that same storage device 120 using submission queues. Load modules 325 may also consider the size of the request and the availability of submission queues to balance storage devices 120 .

In the above discussion, how data is stored in storage devices 120 has not been discussed. In some embodiments of the present disclosure, storage devices 120 may pre-load data, and table 330 may be prepared in advance. In other embodiments of the present disclosure, the applications can request data to be written to storage devices 120, and the manager 330 can select which storage device(s) 120 to write the data to. (Table 330 is updated accordingly).

It would be desirable for data to be accessed approximately equally on all storage devices 120 of the I/O processing/storage pool 315 (this function may be described as load balancing). However, depending on how data is stored in storage devices 120 and what applications are requesting data from storage devices 120, the loads on each storage device 120 may not be balanced. For example, although the storage devices 120-1 and 120-2 may store the same amount of data in terms of size, for example, 80% of the I/O requests are data stored in the storage device 120-1. may occur (and only 20% of the I/O requests request data stored in storage device 120-2). In such a situation, unbalanced loads may result in a higher than desirable latency to access data from storage device 120-1.

To accommodate such situations, the I/O processing/storage pool 315 may include a migration module (not shown in FIG. 3). The migration module may manage the movement of data between storage devices 120 to achieve a desired balance. For example, some data may be migrated from storage device 120-1 to storage device 120-2 to balance how much data is requested from each storage device.

There are other reasons data may move between storage devices 120 . While read load balancing may be an important goal, keeping the capacities of the storage devices roughly balanced (e.g., new data by other applications that may be distributed across the storage devices 120) ) can also be important. Alternatively, some data may be accessed frequently enough or considered sufficiently important to justify the data being stored on multiple storage devices 120 to provide redundancy.

Regardless of the reason the data is being migrated (eg, storage capacity balancing, read load balancing, or redundancy), the migration tool can update the table 330 to reflect those changes. That is, when data is migrated from the storage device 120-1 to the storage device 120-2, the table 330 may be updated to reflect the migration of such data.

FIG. 4 illustrates details of input/output (I/O) requests issued to the multiprocessing system 135 of FIG. 1 according to embodiments of the present disclosure. In FIG. 4 , an I/O request 405 is shown. I/O request 405 is shown as including an identifier 410 and vectors 415-1 through 415-7 (which may be referred to collectively as vectors 415). For example, vectors 415 may include 64 data points, but embodiments of the present disclosure may include any number of data points per vector. Identifier 410 may be an identifier of the requested data from multipurpose system 135 of FIG. 1 . For example, identifier 410 may be a logical address used by an application running on processor 110 of FIG. 1 , but embodiments of the present disclosure may use any desired identifier of the data.

Vectors 415 can identify specific vectors from the data of interest. As described above, DLRM queries can use data (embedding vectors) from embedding tables having large sizes (up to hundreds of GB or TB in size). However, the queries may only depend on specific data in the table, and reading the entire table may take a long time compared to the amount of data actually needed. Instead, I/O request 405 may include vectors 415, which may identify specific vectors of interest in the embedding table, and all other vectors may be ignored. 4 shows five vectors 415 in the I/O request 405, embodiments of the present disclosure may include any number of vectors.

Also, when the I/O request 405 includes the vectors 415, the I/O scheduler 305 of FIG. 3 may determine the size of data to be read. For example, the I/O scheduler 305 of FIG. 3 calculates the number of vectors 415 of the I/O request 405 as the number of data points in each vector 415 and the size of each data point in each vector 415. Multiply to determine the number of bytes to read. If each vector 415 contains, for example, 128 data points each requiring '4' bytes, then the size of the data to be read by I/O request 405 is 64 (the vector in I/O request 405 s 415) × 128 (the number of data points in each vector 415) = 32,768 B.

4 shows I/O request 405 as including only identifier 410 and vectors 415, embodiments of the present disclosure may include other data or may remove some of the data shown. For example, instead of including vectors 415, an I/O request 405 is sent to a logical address (in FIG. 4, the logical address is used as identifier 410, but embodiments of the present disclosure and a logical address for data. In this case, the logical address may be separate data included in the I/O request 405) and the number of bytes to read (in this case, the number of bytes to read) The number can be used to determine the size of the I/O request 405). Alternatively, the I/O request 405 may include various tags not shown in FIG. 4 . Embodiments of this disclosure may include any such variations on I/O request 405 .

FIG. 5 shows details of the table 330 of FIG. 3 according to embodiments of the present disclosure. In FIG. 5 , table 330 is shown as including three entries. One entry maps the identifier 410-1 of the first data to the identifier 505-1 of a storage device that stores the corresponding data. Another entry maps the identifier 410-2 of the second data to the identifier 505-2 of a storage device that stores the corresponding data. The third entry maps the identifier 410-3 of the third data to the identifier 505-3 of a storage device that stores the corresponding data. 5 shows table 330 as including three entries, embodiments of the present disclosure may include any number of entries ('0' or greater).

Identifiers 505-1, 505-2, and 505-3 (which may be collectively referred to as identifier 505) are shown identifying specific storage devices by number. Since the actual physical address may be stored in the storage device itself, table 330 may not store the actual physical address. Identifier 305 may be replaced with other information capable of uniquely identifying storage devices. For example, an identifier assigned to storage devices 120 of FIG. 1 during discovery and/or enumeration or a serial number of storage devices 120 of FIG. 1 among other possibilities may be used.

5 suggests that each identifier 410 may be associated with a single unique identifier 505, embodiments of the present disclosure may map identifiers 410 to one or more identifiers 505. there is. If the data associated with identifier 410-1 is stored on multiple storage devices (eg, to provide redundancy), table 330 may reflect this fact. If the data in question is stored on multiple storage devices 120 in FIG. 1, the manager 320 in FIG. 3 assigns the I/O request 405 in FIG. 4 to one or more load modules 325 in FIG. You may have the option to This option may be useful, for example, to balance the loads on the storage devices 120 of FIG. 1 in the I/O processing/storage pool 315 of FIG. 3 . For example, the number of I/O requests 405 in FIG. 4 waiting to be processed by the storage device 120-1 in FIG. If available, the manager 320 of FIG. 3 selects the load modules 325-3 or 325-4 of FIG. 3 to perform the I/O request 405 from the storage device 120-2 of FIG. can

In some embodiments of the present disclosure, the data requested in the I/O request 405 of FIG. 4 may be stored in the single storage device 120 of FIG. 1 . In such embodiments of the present disclosure, identifier 505 may uniquely identify where the data represented by identifier 410 is located. However, in other embodiments of the present disclosure, the data may be distributed across multiple storage devices 120 of FIG. 1 . In such embodiments of the present disclosure, manager 320 of FIG. 3 determines that all data requested in I/O request 405 of FIG. 4 may be distributed across multiple storage devices 120 of FIG. , and can split the I/O request 405 of FIG. 4 into a number of different I/O requests, and send each to the different load modules 325 of FIG. 3 . Since table 330 may identify what data storage device 120 of FIG. 1 is storing, table 330 may include multiple entries for different portions of the data.

As an example, consider again I/O request 405 of FIG. I/O request 405 of FIG. 4 requests data from the five vectors 415 of FIG. The manager 320 of FIG. 3 has the vectors 415-1 and 415-4 of FIG. 4 stored in the storage device 120-1 of FIG. 1 and the vectors 415-2 and 415-4 of FIG. 3 and 415-5) may be determined to be stored in the storage device 120-2 of FIG. In this case, the manager 320 of FIG. 3 transmits one I/O request to the load modules 325-1 or 325-2 of FIG. 3 to read vectors 415-1 and 415-4 of FIG. and other I/O requests may be sent to the load modules 325-3 or 325-4 of FIG. 3 to read the vectors 415-2, 415-3, and 415-5 of FIG.

Also recall that multiple load modules 325 of FIG. 3 may access the same storage device 120 of FIG. 1 . In that case, even if the data to be read is stored only in the storage device 120-1 of FIG. 1, for example, the manager 320 of FIG. 1), and the other to the load module 325-2 of FIG. 3. In this way, even though a single load module 325 of FIG. 3 can handle reading all the data from the storage device 120-1 of FIG. 1, the manager 320 of FIG. Reading data from device 120-1 may be facilitated.

FIG. 6 shows details of the I/O scheduler 305 of FIG. 3 in accordance with embodiments of the present disclosure. In FIG. 5 , the I/O scheduler 305 may include a size calculator 605 , a threshold 610 , a comparator 615 , and a queue selector 620 . As described above with reference to FIGS. 3 and 4, the I/O scheduler 305 uses the size of the I/O request 405 of FIG. 4 to place the I/O request 405 in FIG. 310) can be selected. The size calculator 605 of FIG. 6 may determine the size of the I/O request 405 of FIG. 4 . The size calculator 605 calculates, among other data, the number of bytes to be read from the data according to the input/output request 405 of FIG. 4 or the number of data points of each vector 415 of FIG. 4 combined with the number of bytes per data pointer. The size of the I/O request 405 of FIG. 4 can be calculated using the number of vectors 415 of FIG. 4 .

When the size of the I/O request 405 of FIG. 4 is determined by the size calculator 605, the comparator 615 may compare the I/O request 405 of FIG. 4 with the threshold 610. Threshold 610 can be any desirable threshold comparable to the size of I/O request 405 of FIG. 4, and queue selector 620 uses this information when selecting among queues 310 of FIG. I/O requests 405 of 4 can be placed. If the size of the I/O request 405 of FIG. 4 is smaller than the threshold 610 according to the comparator 615, as determined by the size calculator 605, the queue selector 620 selects one queue ( 310) to place the I/O request 405 of FIG. Otherwise, the queue selector 620 may select another queue 310 of FIG. 3 to place the I/O request 405 of FIG. 4 .

Although FIG. 6 shows one threshold 610 according to embodiments of the present disclosure, embodiments of the present disclosure may include any number of thresholds 610, and comparator 615 is shown in FIG. until a maximum threshold 610 is identified that is less than the size of the I/O request 405 of (or, alternatively, a minimum threshold 610 that is greater than the size of the I/O request 405 of FIG. 4 is identified); The size of the I/O request 405 of FIG. 4 and each threshold 610 may be compared. The queue selector 620 can then use this information to place the I/O request 405 of FIG. 4 by selecting the queue 310 of FIG. 3 .

For example, as described above with reference to FIG. 3, queue 310-1 of FIG. 3 may be used to store I/O requests to retrieve data, for example, no greater than 128 embedding vectors. . Queue 310-2 of FIG. 3 may be used to store I/O requests for retrieving data greater than, for example, 128 embedding vectors but less than, for example, 512 embedding vectors. Queue 310-3 of FIG. 3 may be used to store I/O requests for retrieving data larger than 512 embedding vectors, for example. In this example, two thresholds 610 may be used, one at 128 embedding vectors and the other at 512 embedding vectors. Thus, in some embodiments of the present disclosure, the number of queues 310 in FIG. 3 may be one greater than the number of thresholds 610 (thresholds 610 may be between pairs of queues 310 in FIG. 3 ). They serve as dividing lines).

FIG. 7 shows details of the load module 325 of FIG. 3 according to embodiments of the present disclosure. As discussed above with reference to FIG. 4, the I/O request 405 of FIG. 4 may identify specific vectors 415 of FIG. 4 to be read from the embedding table. Since the number of vectors 415 of FIG. 4 in the I/O request 405 of FIG. 4 may be small compared to the number of vectors of data, most of the data may be ignored. Data in storage devices 120 of FIG. 1 can therefore be considered “sparse” in that most values can be ignored for purposes of satisfying I/O requests 405 of FIG. 4 . there is.

When load module 325 is a sparse length sum (SLS) load module, load module 325 can read certain vectors 415 of FIG. 4 identified in I/O request 405 of FIG. They can be added together to create a single vector. The sum of the identified vectors 415 of FIG. 4 may then be returned as the data requested by the I/O request 405 of FIG. 4 .

To operate as the SLS load module 325, the load module 325 may include a reader 750 and an adder 710. The reader 705 can read the particular vectors 415 of FIG. 4 identified in the I/O request 405 of FIG. 4 from the storage device 120 of FIG. 1 . Note that in some embodiments of the present disclosure, reader 705 may somehow access data from storage device 120 of FIG. 1 . In other embodiments of the present disclosure, reader 705 may issue appropriate commands to storage device 120 of FIG. 1 , and storage device 120 may return the data to load module 325 . Next, the adder 710 may add the vectors retrieved from the storage device 120 of FIG. 1 to generate data returned in response to the input/output request 405 of FIG. 4 .

FIG. 8 shows details of computing system 140 of FIG. 1 in accordance with embodiments of the present disclosure. In FIG. 8 , computing system 140 may receive operation request 805 , which is retrieved by multiprocessing system 135 of FIG. 1 in response to input/output request 405 of FIG. 4 . It may be a request to process the data to be processed. Computing system 140 includes computation scheduler 810, queues 815-1 and 815-2 (which may be referred to collectively as queues 815), and processing elements 820-1, 820-2, and 820-3) (which may be referred to collectively as processing elements 820).

Operation scheduler 810 can place operation request 805 in one of queues 815 based on the workload of operation request 805 . For example, the operation request 805 may include resources provided by only one of the processing elements 820 that may determine the queue 815 to which the operation request 805 should be placed. Operation scheduler 810 may also consider how busy processing elements 820 are in assigning operation requests 805 to processing elements 820, as described below.

In some embodiments of the present disclosure, queues 815 may be FIFO queues. In other embodiments of the present disclosure, other types of queues 815 may be used.

Processing element 820 may then remove operation request 805 from queues 815 and process the request. Processing element 820 may be any desired type of processing element. For example, the processing element 820 may be a central processing unit (CPU), a graphics processing unit (GPU), a general purpose GPU (GPGPU), a neural processing unit (NPU), a tensor processing unit (TPU), or a field programmable unit (FPGA). It could be an accelerator such as a Gate Array or an Application-Specific Integrated Circuit (ASIC) among other possibilities. Also, for elements that include multiple cores (eg, a multi-core CPU), each core may be considered a separate processing element 820 .

In some embodiments of the present disclosure, each processing element 820 may have its own queue 815 from which it may receive operation requests to process. That is, processing element 820 may process only specifically assigned operation requests and may ignore operation requests assigned to other processing elements. In other embodiments of the present disclosure, two or more processing elements may share a queue. For example, as shown in FIG. 8, processing elements 820-1 and 820-2 may both receive operation requests via queue 815-1, while processing element 820-3 may receive operation requests through the queue 820-3. When a processing element has completed processing of an operation request, the processing element may then check the appropriate queue to see if there are other operation requests waiting to be processed. If the processing element finds a pending operation request that it can process, the processing element may begin processing the operation request. Otherwise the processing element may be idle.

In some embodiments of the present disclosure, a processing element may seek operation requests in multiple queues 815 . For example, processing element 820-3 can process any computational request that processing elements 820-1 and 820-2 can process, but can also process some additional computational requests. In such a situation, processing element 820-3 may retrieve operation requests from queue 815-2 as long as queue 815-2 has operation requests waiting to be processed. If queue 815-2 is empty, processing element 820-3 may retrieve the operation request from queue 815-1.

Computing system 140 may also include a ready queue 825 . Processing elements 820 may use ready queue 825 to notify operation scheduler 810 when processing elements 820 have finished processing an operation request. In this way, operation scheduler 810 can track how busy processing elements 820 are. For example, consider the situation where operation scheduler 810 receives operation request 805, and assume that each processing element 820 has its own queue 815. Operation scheduler 810 may determine that one of processing elements 820-1 or 820-2 is capable of processing the operation request. Without any information about how busy processing elements 820-1 or 920-2 are, operation scheduler 810 randomly sends operation requests 805 to queues associated with processing elements 820-1 or 920-2. can be assigned to However, if operation scheduler 810 receives information via ready queue 825 that processing element 820-2 has completed its most recent operation request (and is therefore currently idle), operation scheduler 810 may perform operation Request 805 can be assigned to processing element 820-2 without having to guess which processing elements 820-1 or 820-2 have a light workload.

In a similar fashion, processing elements 820-1 or 820-2 may be the preferred processing element for processing operation request 805, but both processing elements 820-1 and 820-2 are currently busy. busy, and processing element 820-3 is currently idle, operation scheduler 810 determines operation request 805 even if processing element 820-3 is less desirable to process operation request 805. ), the processing element 820-3 may be scheduled.

The operation request 805 may include a tag identifying the operation request. Alternatively, operation scheduler 810 may identify operation request 805 by assigning a tag to operation request 805 . Processing elements 820 may use these tags in ready queue 825 to inform operation scheduler 810 what operation requests have been completed. In this way, operation scheduler 810 provides information about processing elements 820 (by comparing which operation requests each processing element 820 has processed and which operation requests are scheduled for each processing element 820). You can keep a rough idea of pending workloads for .

FIG. 9 depicts a flowchart of an exemplary procedure for processing an I/O request of FIG. 4 using the multiprocessing system 135 of FIG. 1 in accordance with embodiments of the present disclosure. In FIG. 9 , at block 905 , the I/O scheduler 305 of FIG. 3 may receive the I/O request 405 of FIG. 4 . At block 910, the I/O request 405 of FIG. 4 may be passed from the I/O scheduler 305 of FIG. 3 to the load module 325 of FIG. Finally, at block 915, the load module 325 of FIG. 3 may read data from the storage device 120 of FIG. 1 based on the I/O request 405 of FIG.

FIG. 10 depicts an alternative flow diagram of an exemplary procedure for processing the I/O request of FIG. 4 using the multiprocessing system of FIG. 1 in accordance with embodiments of the present disclosure. In FIG. 10, blocks similar to those of FIG. 9 may be assigned using the same figure reference numerals. In FIG. 10 , at block 905 , the I/O scheduler 305 of FIG. 3 may receive the I/O request 405 of FIG. 4 . At block 1005, size calculator 605 of FIG. 6 may determine the size of I/O request 405 of FIG. At block 1010, the queue selector 620 of FIG. 6 may select the queue 310 of FIG. 3 for the I/O request 405 of FIG. At block 1015, the I/O selector 305 of FIG. 3 may place the I/O request 405 of FIG. 4 to the queue 310 of FIG. 3 selected by the queue selector 620 of FIG. Finally at block 915 the load module 325 of FIG. 3 may read data from the storage device 120 of FIG. 1 based on the I/O request 405 of FIG. 4 .

FIG. 11 shows a flow diagram of an exemplary procedure for using priority queuing for the I/O scheduler 305 of FIG. 3 to queue the I/O requests 405 of FIG. 4 in accordance with embodiments of the present disclosure. In FIG. 11, at block 1105, the I/O scheduler 305 of FIG. 3 bases the I/O request 405 of FIG. 4 on the size of the I/O request 405 of FIG. 4 (which may be determined by size calculator 605 of FIG. 5). A priority tag for 405 can be determined. At block 1110, the I/O scheduler 305 of FIG. 3 may associate a priority tag with the I/O request 405 of FIG. 3 in the queue 310 of FIG.

FIG. 12 illustrates a flow diagram of an exemplary procedure for the manager 320 of FIG. 3 to assign the I/O request 405 of FIG. 4 to the load module 325 of FIG. 3 according to embodiments of the present disclosure. In FIG. 12 , at block 1205 the manager 320 of FIG. 3 may retrieve the I/O request 405 of FIG. 4 from the queue 310 of FIG. 3 . At block 1210, the manager 320 of FIG. 3 can identify the storage device 120 of FIG. 1 that stores the data. This may involve mapping the FIG. 4 identifier 410 of the data to the FIG. 3 identifier of the storage device storing the data using the table 330 of FIG. 3 , for example, as shown in block 1215 . can At block 1220, the manager 320 of FIG. 3 can identify the load module 325 of FIG. 3 that can access the storage device 120 of FIG. Finally, at block 1225, the manager 320 of FIG. 3 may send the I/O request 405 of FIG. 4 to the load module 325 of FIG.

FIG. 13 illustrates a flowchart of an exemplary procedure for the load module 325 of FIG. 3 to read data from the storage device of FIG. 1 in accordance with embodiments of the present disclosure. In FIG. 13 , at block 1305 , reader 705 of FIG. 7 can read vectors 415 of FIG. 5 from storage 120 of FIG. 1 . At block 1310, adder 710 of FIG. 7 may add vectors 415 of FIG. Finally, at block 1315, the multiprocessing system 135 of FIG. 1 transfers the data to the operation scheduler 810 of FIG. 8 of the processing system 140 of FIG. 1 for use in processing the operation request 805 of FIG. can be sent to

FIG. 14 depicts a flow diagram of an exemplary procedure for processing the computation request of FIG. 8 by computing system 140 of FIG. 1 in accordance with embodiments of the present disclosure. In FIG. 14, at block 1405, operation scheduler 810 of FIG. 8 uses data read by multiple processing system 135 of FIG. 1 in response to I/O request 405 of FIG. 4 to request operation of FIG. 4 The processing of 805 can be scheduled. At block 1410 , processing element 820 of FIG. 8 may receive operation request 805 of FIG. 8 from operation scheduler 810 of FIG. 8 . For example, at block 1405 the operation scheduler 810 of FIG. 8 may place the operation request 805 of FIG. 8 into the queue 815 of FIG. 8 and at block 1410 the processing element 820 of FIG. The operation request 805 of FIG. 8 may be retrieved from the queue 815 of FIG. 8 .

At block 1415, the processing element 820 of FIG. 8 may process the operation request 805 of FIG. Finally, at block 1420, processing element 820 of FIG. 8 may inform operation scheduler 810 of FIG. 8 that processing of operation request 805 of FIG. 8 is complete. For example, the processing element 820 of FIG. 8 places information in the ready queue 825 of FIG. 8 to the operation scheduler 810 of FIG. 805) processing may be completed. In some embodiments of the present disclosure, block 1420 (e.g., only one operation request 805 of FIG. 8 is scheduled to processing element 820 of FIG. 8 and operation scheduler 810 of FIG. If it is known how long processing element 820 of FIG. 8 will take to process 8's operation request 805), it can be omitted as shown by dashed line 1425.

15A and 15B are flowcharts of an exemplary procedure in which operation scheduler 810 of FIG. 8 arranges processing element 820 to process operation request 805 of FIG. 8, in accordance with embodiments of the present disclosure. show In FIG. 15A , at block 1505 , operation scheduler 810 of FIG. 8 may select processing element 820 of FIG. 8 to process operation request 805 of FIG. 8 . At block 1510, the operation scheduler 810 of FIG. 8 may send the operation request 805 of FIG. 8 to the processing element 820 of FIG. Alternatively, at block 1515, operation scheduler 810 of FIG. 8 may assign operation request 805 of FIG. 8 to queue 815 of FIG.

As an alternative to block 1505, at block 1520 (FIG. 15B), operation scheduler 810 of FIG. 8 will identify the type of processing element 820 of FIG. 8 suitable for processing operation request 805 of FIG. can At block 1525 the operation scheduler 810 of FIG. 8 may assign the operation request 805 of FIG. 8 to the queue 815 of FIG. 8 appropriate for the type of processing element 820 of FIG.

As another alternative to block 1505, at block 1530, operation scheduler 810 of FIG. 8 may determine the workload of operation request 805 of FIG. At block 1535, the operation scheduler 810 of FIG. 8 may assign the operation request 805 of FIG. 8 to the queue 815 of FIG. 8 based on the workload of the operation request 805 of FIG.

9-15B , some embodiments of the present disclosure are shown. However, those skilled in the art will recognize that other embodiments of the present disclosure are also possible by changing the order of blocks, omitting blocks, or including connections not shown in the figures. All such variations of the flowcharts, whether or not explicitly described, are considered embodiments of the present disclosure.

Embodiments of the present disclosure include multiple processing systems. The multiprocessing system may load data from an input/output processing/storage pool using input/output (I/O) requests, which may be based on data being processed using operation requests. Data may be retrieved using load modules associated with the storage devices of the I/O processing/storage pool. Using multiple storage devices provides technical advantages over storing data in a single storage device by utilizing parallel data access from multiple storage devices to achieve low latency when reading the data.

Different I/O requests can be queued to different queues. Using multiple queues provides a technical advantage in that I/O requests that may contain large or small amounts of data are not delayed by multiple I/O requests of different data sizes.

Data retrieved by the multi-processing system may be provided to the computing system. The computing system can use the data to schedule computational requests. Since the computing system can use different queues based on workloads of computational requests, it is technical to meet the number of queries per second (QPS) promised by service level agreement (SLA). advantage can be gained.

Deep Learning Recommendation Model (DLRM) workloads can be I/O intensive. To meet their SLA requirements, dynamic random access memory (DRAM) may be needed to store large embedding tables (up to 100 GB or more). Such large amounts of DRAM can be expensive.

For small query sizes, SLAs can be met by using solid-state drives (SSDs) with user-space drivers to store the embedding tables. However, for reasonably large query sizes (256 and above), it can be difficult to meet the above SLA with a single SSD. Furthermore, when using a single SSD, it may be impossible to achieve a high QPS by executing queries in parallel due to potential I/O bottlenecks in the SSD.

A query scheduler based on operations using multiple SSDs may have low efficiency and load balancing issues (some SSDs process a higher percentage of queries, others process a lower percentage of queries). .). SSDs that process a large number of queries may not have enough I/O to process the queries (similar to the single SSD model), and SSDs that process a small number of queries may be underutilized.

Embodiments of the present disclosure may include a multi-processing and multi-SSD system with an I/O scheduler to achieve high QPS and low latency. The schedule embedding table I/O can be used to schedule queues to other I/O queues.

The I/O processing/SSD pool may fetch I/O requests from the I/O queues based on load status and I/O requests for the various SSDs.

Within an I/O process/SSD pool, multiple SSDs can be accessed by a multi-process user-space non-volatile memory express (UNVMe) driver/application programming interface (API) that uses multiple active threads to satisfy I/O requests. there is.

A second level computation scheduler can be used to further optimize latency and QPS based on computation intensity.

The following discussion is intended to provide a general description of a suitable apparatus or apparatuses in which certain aspects of the present disclosure may be implemented. The device or devices may act at least in part from existing input devices such as keyboards, mice, etc., as well as by instructions received from another device, interaction with a virtual reality (VR) environment, biofeedback, or other input signal. It can be controlled by the input of

As used herein, the term “device” is intended to broadly include a single device, a virtual device, or a system of communicatively coupled devices, virtual devices, or devices operating together. Exemplary devices include computing devices such as personal computers, workstations, servers, portable computers, portable devices, phones, tablets, etc., as well as personal or public transportation such as cars, trains, taxis, etc. Includes transportation devices.

The device or devices may include embedded controllers such as programmable or non-programmable logic devices or arrays, Application Specific Integrated Circuits (ASICs), embedded computers, smart cards, and the like. The device or devices may utilize one or more connections, such as a network interface, modem, or other communicative connection, to one or more remote devices. Devices may be interconnected by way of physical and/or logical networks, such as intranets, the Internet, local area networks (LANs), wide area networks (WANs), and the like. Those skilled in the art will understand that network communications include radio frequency (RF), satellite, microwave, Institute of Electrical and Electronics Engineers (IEEE) 802.11, Bluetooth®, optical, infrared, cable, laser, and the like. It will be appreciated that a variety of wired and/or wireless short or long range carriers and protocols may be utilized.

Embodiments of the present disclosure describe functions, procedures, data structures, and application programs that, when accessed by a device, result in the device performing tasks, defining abstract data types, or defining low-level hardware contexts. It may refer to or be described in conjunction with associated data including the like. The associated data may include, for example, volatile and/or non-volatile memory (e.g. RAM, ROM, etc.) or other storage devices and hard drives, floppy disks, optical storage devices, tapes, flash memory, memory sticks files, digital videodisks, biological storage devices, and the like. Associated data may be carried in the form of packets, serial data, parallel data, propagated signals, etc., over transmission environments including physical and/or logical networks, and may be used in compressed or encrypted form. Associated data may be used in a distributed environment and may be stored locally and/or remotely for device access.

Embodiments of the present disclosure may include a tangible, non-transitory machine-readable medium containing instructions executable by one or more processors to perform the elements of the present disclosure described herein. may contain directives. The various methods of operation described above may be performed by any suitable means capable of performing the operations, such as various hardware and/or software component(s), circuits, and/or module(s). The software may include an ordered listing of executable instructions to implement logical functions, for use by or in conjunction with an instruction execution system or device, such as a single or multi-core processor or system containing processors. may be embodied in any “processor readable medium”.

Blocks or steps and functions of a method or algorithm described in connection with the embodiments disclosed herein may be implemented directly in hardware, in a software module executed by a processor, or in a combination of the two. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code in a tangible, non-transitory computer readable medium. A software module may include RAM (Random Access Memory), flash memory, ROM (Read Only Memory), EPROM (Electrically Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), registers, hard disk, removable disk, CD ROM, or It may reside on any other form of storage medium known in .

Although the principles of the present disclosure have been described and illustrated with reference to the illustrated embodiments, it will be appreciated that the illustrated embodiments may be modified in arrangement and detail and combined in any desired manner without departing from these principles. And while the foregoing discussion has focused on specific embodiments, other configurations are contemplated. In particular, despite phrases such as “according to one embodiment of the present disclosure” and the like used herein, such phrases are generally intended to refer to possible embodiments and are not intended to limit the present disclosure to specific embodiment configurations. is not As used herein, these terms may refer to the same or different embodiments that are combinable into other embodiments. The foregoing exemplary embodiments should not be construed as limiting its disclosure. Although several embodiments have been described, those skilled in the art will readily appreciate that many modifications to such embodiments are possible without materially departing from the novel teachings and advantages of the present disclosure. Accordingly, all such modifications are intended to be included within the scope of this disclosure as defined in the claims.

Embodiments of the present disclosure may extend to the following statements without limitation.

Statement 1. One embodiment of the present disclosure:

a storage device that stores data;

a load module that reads the data from the storage device based at least in part on input/output (I/O) requests; and

and a scheduler that receives the I/O requests and directs the I/O requests to the load module based at least in part on the size of the I/O requests.

Statement 2. An embodiment of the present disclosure includes the system according to statement 1, wherein the scheduler is configured to forward the I/O request to a queue based at least in part on a size of the I/O request.

Statement 3. An embodiment of the present disclosure includes the system according to statement 2, further comprising a manager to retrieve the I/O request from the queue and assign the I/O request to the load module.

Statement 4. One embodiment of the present disclosure includes the system according to Statement 3, wherein the system comprises:

a second storage device that stores second data; and

a second load module for reading the second data from the second storage device;

The manager is configured to assign the I/O request to the load module based at least in part on the I/O request requesting the data.

Statement 5. One embodiment of the present disclosure includes the system according to Statement 4;

The input/output request includes a first identifier of the data, and the manager includes a table mapping the first identifier of the data to a second identifier of the storage device.

Statement 6. An embodiment of the present disclosure includes the system according to Statement 3, wherein the system includes a second load module to read the data from the storage device.

Statement 7. An embodiment of the present disclosure includes the system according to Statement 6, wherein the manager is configured to select the load module to process the I/O request.

Statement 8. An embodiment of the present disclosure includes the system according to Statement 3;

The queue comprises a first in first out (FIFO) queue, and the manager is configured to access the input/output requests from the head of the queue.

Statement 9. One embodiment of the present disclosure includes the system according to Statement 3;

The queue includes a priority queue;

the scheduler is configured to associate the I/O request with a priority tag based at least in part on the size of the I/O request; and

The manager is configured to access the input/output requests from the queue based at least in part on the priority tag.

Statement 10. An embodiment of the present disclosure includes the system according to Statement 3;

the scheduler includes a threshold, and

The scheduler is configured to place the I/O request into the queue based at least in part on the size of the I/O request being less than the threshold.

Statement 11. An embodiment of the present disclosure includes the system according to Statement 10;

The system further comprises a second queue, and

The scheduler is configured to place a second I/O request into the second queue based at least in part on the size of the second I/O request exceeding the threshold.

Statement 12. An embodiment of the present disclosure includes the system according to Statement 11, wherein the manager retrieves the I/O request from the queue using round robin access, and retrieves the I/O request from the second queue. and retrieve the second input/output request.

Statement 13. An embodiment of the present disclosure includes the system according to Statement 1, wherein the load module comprises:

a reader that reads the first vector and the second vector of data from the storage device; and

and an adder for adding the first vector and the second vector.

Statement 14. An embodiment of the present disclosure includes the system according to Statement 1, wherein the load module is configured to transmit the data to a second scheduler.

Statement 15. One embodiment of the present disclosure includes the system according to Statement 14, the system comprising:

processing element; and

and a second scheduler configured to schedule processing of operation requests by the processing element, wherein the operation requests use the data.

Statement 16. An embodiment of the present disclosure includes the system according to statement 14, wherein the second scheduler is configured to assign the operation request to a queue.

Statement 17. An embodiment of this disclosure includes the system according to Statement 16, wherein the second scheduler is configured to assign the operation request to the queue based at least in part on a workload of the operation request.

Statement 18. An embodiment of the present disclosure includes the system according to Statement 16, wherein the processing element is configured to retrieve the operation request from the queue.

Statement 19. An embodiment of the present disclosure includes the system according to Statement 16;

the queue associated with the processing element; and

and a second scheduler configured to assign the second operation request to a second queue associated with the second processing element.

Statement 20. An embodiment of the present disclosure includes the system according to Statement 16, wherein the queue is associated with the type of processing element.

Statement 21. An embodiment of the present disclosure includes the system according to Statement 16, wherein the processing element is configured to return a completion status to the second scheduler.

Statement 22. An embodiment of the present disclosure includes the system according to Statement 21, wherein the processing element is configured to place the completion status in a second queue.

Statement 23. An embodiment of the present disclosure includes the system according to statement 22, wherein the second scheduler is configured to retrieve the completion status from the second queue.

Statement 24. An embodiment of the present disclosure includes the system according to Statement 22, wherein a second processing element is configured to place a second completion state into the second queue.

Statement 25. One embodiment of the present disclosure:

Receiving an input/output (I/O) request from a scheduler;

forwarding the I/O request from the scheduler to a load module based at least in part on the size of the I/O request; and

and reading data from a storage device based at least in part on the I/O request by the load module.

Statement 26. One embodiment of the present disclosure includes the method according to Statement 25;

Passing the I/O request from the scheduler to the load module based at least in part on the size of the I/O request comprises:

determining the size of the input/output request by the scheduler;

identifying, by the scheduler, a queue based at least in part on the size of the I/O request; and

placing the I/O request into the queue by the scheduler.

Statement 27. One embodiment of the present disclosure includes the method according to Statement 26;

The step of reading the data from the storage device by the load module,

retrieving the input/output request from the queue by a manager; and

and transmitting the input/output request from the manager to the load module.

Statement 28. One embodiment of the present disclosure includes the method according to Statement 27, wherein sending the I/O request to the load module comprises:

identifying the storage device storing the data by the administrator; and

and identifying the load module based at least in part on the load module accessing the storage device by the manager.

Statement 29. One embodiment of the present disclosure includes the method according to Statement 28;

The identifying of the storage device storing the data by the manager includes mapping a first identifier of the input/output request to a second identifier of the storage device.

Statement 30. One embodiment of the present disclosure includes the method according to Statement 28, wherein identifying the storage device storing the data by the administrator comprises: identifying the storage device and the second storage device by the administrator; and identifying the storage device that stores the data from.

Statement 31. One embodiment of the present disclosure includes the method according to Statement 27;

The queue comprises a first-in-first-out (FIFO) queue;

Placing the input/output request to the queue by the scheduler comprises:

determining a priority tag for the I/O request based at least in part on the size of the I/O request; and

associating the priority tag with the input/output request of the queue.

Statement 32. One embodiment of the present disclosure includes the method according to Statement 31;

Retrieving the I/O request from the queue by the manager includes retrieving the I/O request from the queue by the manager based at least in part on the priority tag.

Statement 33. One embodiment of the present disclosure includes the method according to Statement 27;

identifying the queue based at least in part on the size of the I/O request by the scheduler;

and identifying, by the scheduler, the queue based at least in part on the size of the input/output request being less than a threshold.

Statement 34. One embodiment of the present disclosure includes the method according to Statement 33;

identifying the queue based at least in part on the size of the I/O request by the scheduler;

and identifying, by the scheduler, a second queue based at least in part on the size of the second I/O request being greater than a threshold.

Statement 35. One embodiment of the present disclosure includes the method according to Statement 34;

The step of retrieving the input/output request from the queue by the manager,

and retrieving the I/O request from the queue using round robin access, and retrieving the second I/O request from the second queue.

Statement 36. One embodiment of the present disclosure includes the method according to Statement 25;

The step of reading the data from the storage device by the load module,

reading a first vector from the storage device;

reading a second vector from the storage device; and

and generating the data by adding the first vector and the second vector.

Statement 37. One embodiment of the present disclosure includes the method according to Statement 25;

and transmitting the data from the load module to a second scheduler.

Statement 38. One embodiment of the present disclosure includes the method according to Statement 37;

Scheduling a processing element to process a computational request using the data.

Statement 39. One embodiment of the present disclosure includes the method according to Statement 38;

Scheduling the processing element to process the operation request using the data comprises:

and assigning the operation request to a queue.

Statement 40. An embodiment of the present disclosure includes the method according to Statement 39;

The step of assigning the operation request to the queue,

and assigning the operation request to the queue based at least in part on the workload of the operation request.

Statement 41. One embodiment of the present disclosure includes the method according to Statement 40;

retrieving the operation request from the queue by the processing element; and

and processing the operation request using the data by the processing element.

Statement 42. An embodiment of the present disclosure includes the method according to Statement 41;

and returning a completion status from the processing element in a second queue to the second scheduler.

Statement 43. One embodiment of the present disclosure includes the method according to Statement 39;

and retrieving the completion status from the second queue by the second scheduler.

Statement 44. One embodiment of the present disclosure includes the method according to Statement 43;

and retrieving a second completion status from the second queue by the second scheduler, the second completion status being placed in the second queue by a second processing element.

Statement 45. An embodiment of this disclosure includes the method according to Statement 39, wherein assigning the operation request to the queue comprises assigning the operation request to the queue associated with the processing element. .

Statement 46. One embodiment of the present disclosure includes the method according to Statement 39;

Scheduling the processing element to process the operation request using the data comprises:

further comprising identifying the type of processing element based at least in part on the operation request;

Assigning the operation request to the queue includes assigning the operation request to the queue associated with the type of processing element.

Statement 47. One embodiment of this disclosure includes an article comprising a non-transitory storage medium, wherein the non-transitory storage medium, when executed by a device:

receiving input/output (I/O) requests from the scheduler;

forwarding the I/O request from the scheduler to a load module based at least in part on the size of the I/O request; and

stores instructions that result in reading data from the storage device by the load module based at least in part on the I/O request.

Statement 48. An embodiment of this disclosure includes the article according to Statement 47, wherein forwarding the I/O request from the scheduler to the load module based at least in part on the size of the I/O request comprises:

determining the size of the I/O request by the scheduler;

identifying a queue based at least in part on the size of the I/O request by the scheduler; and

and placing the I/O request into the queue by the scheduler.

Statement 49. One embodiment of this disclosure includes the article according to Statement 48, wherein reading the data from the storage device by the load module comprises:

retrieving the I/O request from the queue by a manager; and

and transmitting the input/output request from the manager to the load module.

Statement 50. One embodiment of this disclosure includes the article according to Statement 49, wherein sending the I/O request to the load module comprises:

identifying the storage device storing the data by the manager; and

and identifying the load module based at least in part on the load module having access to the storage device by the administrator.

Statement 51. One embodiment of this disclosure includes the article according to Statement 50, wherein identifying the storage device storing the data by the administrator includes:

and mapping the first identifier of the input/output request to the second identifier of the storage device.

Statement 52. One embodiment of this disclosure includes the article according to Statement 50, wherein identifying the storage device storing the data by the administrator includes:

and identifying the storage device storing the data from the storage device and a second storage device by the administrator.

Statement 53. An embodiment of the present disclosure includes the article according to Statement 49;

the queue comprises a first-in-first-out (FIFO) queue;

Placing the I/O request to the queue by the scheduler comprises:

determining a priority tag for the I/O request based at least in part on the size of the I/O request; and

and associating the priority tag with the input/output request of the queue.

Statement 54. One embodiment of this disclosure includes the article according to Statement 53, wherein retrieving the I/O request from the queue by the manager comprises:

and retrieving the I/O request from the queue by the manager based at least in part on the priority tag.

Statement 55. One embodiment of this disclosure includes the article according to statement 49, wherein identifying the queue based at least in part on the size of the I/O request by the scheduler comprises:

and identifying the queue based at least in part on the size of the I/O request being less than a threshold by the scheduler.

Statement 56. One embodiment of this disclosure includes the article according to Statement 55, wherein identifying the queue based at least in part on the size of the I/O request by the scheduler comprises:

and identifying, by the scheduler, a second queue based at least in part on the size of a second I/O request that is greater than a threshold.

Statement 57. One embodiment of this disclosure includes the article according to Statement 56, wherein retrieving the I/O request from the queue by the manager comprises:

and retrieving the I/O request from the queue and the second I/O request from the second queue using round robin access.

Statement 58. One embodiment of this disclosure includes the article according to Statement 47, wherein reading the data from the storage device by the load module comprises:

reading a first vector from the storage device;

reading a second vector from the storage device; and

and generating the data by adding the first vector and the second vector.

Statement 59. An embodiment of this disclosure includes the article according to Statement 47, wherein the non-transitory storage medium, when executed by the device,:

Stores additional directives that result in transferring the data from the load module to the second scheduler.

Statement 60. An embodiment of this disclosure includes the article according to Statement 59, wherein the non-transitory storage medium, when executed by the device,:

The data is used to store additional directives that result in scheduling a processing element to process a computational request.

Statement 61. An embodiment of this disclosure includes the article according to statement 60, wherein scheduling the processing element to process the computational request using the data includes:

and assigning the operation request to a queue.

Statement 62. One embodiment of this disclosure includes the item according to statement 61, wherein assigning the operation request to the queue comprises:

and assigning the operation request to the queue based at least in part on the workload of the operation request.

Statement 63. An embodiment of this disclosure includes the article according to Statement 62, wherein the non-transitory storage medium, when executed by the device,:

retrieving the operation request from the queue by the processing element; and

Stores additional directives that result from processing the operation request using the data by the processing element.

Statement 64. An embodiment of this disclosure includes the article according to Statement 63, wherein the non-transitory storage medium, when executed by the device,:

Stores additional directives that result in returning a completion status from the processing element in a second queue to the second scheduler.

Statement 65. An embodiment of this disclosure includes the article according to Statement 61, wherein the non-transitory storage medium, when executed by the device,:

Store additional directives that result in retrieving the completion status from the second queue by the second scheduler.

Statement 66. An embodiment of this disclosure includes the article according to Statement 65, wherein the non-transitory storage medium, when executed by the device,:

Retrieve a second completion state from the second queue by the second scheduler, the second completion state storing additional instructions that result in being placed in the second queue by a second processing element.

Statement 67. One embodiment of this disclosure includes the item other than in statement 61, wherein assigning the operation request to the queue comprises:

and assigning the operation request to the queue associated with the processing element.

Statement 68. An embodiment of the present disclosure includes the article according to Statement 61;

scheduling the processing element to process the computational request using the data includes identifying a type of the processing element based at least in part on the computational request;

Assigning the operation request to the queue includes assigning the operation request to the queue associated with the type of processing element.

Consequently, the detailed description and accompanying material, in view of various modifications to the embodiments described herein, are illustrative only and should not be construed as limiting the scope of the present disclosure. Accordingly, what is claimed as this disclosure are all such modifications as may come within the scope and spirit of the following claims and equivalents thereto.

110: processor
115: memory
120-1: storage device
120-2: storage device
125: memory controller
130: device driver
135: multiprocessing system
140: operation system

Claims (10)

a storage device that stores data;
a load module that reads the data from the storage device based at least in part on an I/O request; and
a scheduler to receive the I/O requests and to place the I/O requests into a queue for delivery to the load module based at least in part on the size of the I/O requests being less than a threshold. According to claim 1,
and a manager for retrieving the I/O requests from the queue and assigning the I/O requests to the load modules. The method of claim 2, wherein the system,
a second storage device that stores second data; and
a second load module for reading the second data from the second storage device;
wherein the manager assigns the I/O request to the load module based at least in part on the I/O request requesting the data. According to claim 3,
The input/output request includes a first identifier of the data,
wherein the manager includes a table mapping the first identifier of the data to a second identifier of the storage device. The method of claim 2, wherein the system,
and a second load module to read the data from the storage device. 6. The system of claim 5, wherein the manager selects the load module to process the I/O request. According to claim 2,
The system further comprises a second queue;
wherein the scheduler places the second I/O request into the second queue based at least in part on a size of the second I/O request exceeding the threshold. The method of claim 7, wherein the manager,
A system for retrieving the I/O request from the queue and the second I/O request from the second queue using round robin access. The method of claim 1 , wherein the load module:
a reader that reads the first vector and the second vector of data from the storage device; and
and an adder for adding the first vector and the second vector. 2. The system of claim 1, wherein the load module is configured to transmit the data to a second scheduler.