US20210004229A1 - Open channel vector command execution - Google Patents
Open channel vector command execution Download PDFInfo
- Publication number
- US20210004229A1 US20210004229A1 US16/459,183 US201916459183A US2021004229A1 US 20210004229 A1 US20210004229 A1 US 20210004229A1 US 201916459183 A US201916459183 A US 201916459183A US 2021004229 A1 US2021004229 A1 US 2021004229A1
- Authority
- US
- United States
- Prior art keywords
- vector
- command
- commands
- memory
- controller
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 239000007787 solid Substances 0.000 claims abstract description 8
- 230000015654 memory Effects 0.000 claims description 64
- 230000004044 response Effects 0.000 claims description 6
- 230000005540 biological transmission Effects 0.000 claims description 4
- 238000000034 method Methods 0.000 abstract description 22
- 230000008569 process Effects 0.000 description 16
- 238000004891 communication Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 8
- 238000012545 processing Methods 0.000 description 5
- 238000013403 standard screening design Methods 0.000 description 5
- 238000012546 transfer Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000013507 mapping Methods 0.000 description 3
- 230000008520 organization Effects 0.000 description 3
- 230000003213 activating effect Effects 0.000 description 2
- 238000013519 translation Methods 0.000 description 2
- 238000012937 correction Methods 0.000 description 1
- 238000013506 data mapping Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000007480 spreading Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
- G06F13/14—Handling requests for interconnection or transfer
- G06F13/16—Handling requests for interconnection or transfer for access to memory bus
- G06F13/1668—Details of memory controller
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0659—Command handling arrangements, e.g. command buffers, queues, command scheduling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/30—Arrangements for executing machine instructions, e.g. instruction decode
- G06F9/30003—Arrangements for executing specific machine instructions
- G06F9/30007—Arrangements for executing specific machine instructions to perform operations on data operands
- G06F9/30036—Instructions to perform operations on packed data, e.g. vector, tile or matrix operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/30—Arrangements for executing machine instructions, e.g. instruction decode
- G06F9/30003—Arrangements for executing specific machine instructions
- G06F9/3004—Arrangements for executing specific machine instructions to perform operations on memory
Definitions
- Embodiments of the present disclosure generally relate to systems and methods for executing vector commands on a memory device.
- a solid state drive (SSD) storage device is a data storage device commonly composed of a controller and an SSD memory (e.g., a non-volatile NAND flash).
- the controller sits between one or more host computing systems and the SSD memory.
- the SSD storage device interacts with a host using a flash-translation layer (FTL).
- FTL flash-translation layer
- Open-channel SSDs differ from a traditional SSD in that they expose the internal mapping of the SSD memory structure to the host, allowing the host to perform data placement and data input/output (I/O) scheduling using the physical addresses instead of the traditional logical addresses. Accordingly, the open-channel SSD has the ability to leave certain aspects of the FTL for the host to handle. As the host takes on additional responsibilities that were once relegated to the storage device, additional commands, such as vector commands may be used to enhance the functionality of the SSDs.
- the present disclosure generally relates to a method and apparatus that provide a solid state drive controller that analyzes input/output commands from a host computing device to determine whether the commands include a vector or non-vector command, and is configured to generate a plurality of non-vector commands based on the physical addresses contained in the vector command.
- the disclosure describes a device, comprising a memory and a controller communicatively coupled to the memory, wherein the controller is configured to identify a set of physical addresses in a vector command fetched from a host computing device, wherein each physical address of the set of physical addresses corresponds to a location in the memory.
- the controller is also configured to generate a set of non-vector commands, wherein each non-vector command in the set of non-vector commands corresponds to one of each physical address in the set of physical addresses.
- the controller is also configured to execute the set of non-vector commands.
- the controller is configured to generate a message to the host computing device indicating completion of the vector command in response to the execution of the set of non-vector commands.
- the disclosure describes device, comprising a memory and a controller communicatively coupled to the memory, wherein the controller is configured to receive an indication that the memory has executed a non-vector command, wherein the non-vector command is one of a plurality of non-vector commands corresponding to a vector command received from a host computing device.
- the controller is also configured to determine whether the memory has executed all of the plurality of non-vector commands corresponding to the vector command.
- the controller is also configured to generate a completion message indicating execution of the vector command if the memory has executed each of the plurality of non-vector commands.
- the controller is also configured to transmit the completion message to the host computing device.
- the disclosure describes an apparatus.
- the apparatus includes means for receiving a vector command from a host device, wherein the vector command comprises a set of physical addresses.
- the apparatus include a means for generating a set of non-vector commands, wherein each non-vector command in the set of non-vector commands corresponds to one of each physical address in the set of physical addresses, and wherein each non-vector command in the set of non-vector commands is configured to be executed by a means for storing digital data.
- the apparatus includes a means for generating a message to the host computing device indicating completion of the vector command in response to the execution of the set of non-vector commands by the means for storing digital data.
- FIG. 1 is a block diagram of a memory device communicatively coupled to a host computer system, in accordance with certain aspects of the disclosure.
- FIG. 2 is a block diagram illustrating a logical representation of the internal parallel organization of an open-channel solid state drive (SSD), in accordance with certain aspects of the disclosure.
- SSD solid state drive
- FIG. 3 is a flow diagram illustrating an exemplary process for executing a vector command in accordance with certain aspects of the disclosure.
- FIG. 4 is a flow diagram illustrating an exemplary process for generating a vector command completion message in accordance with certain aspects of the disclosure.
- the present disclosure relates to a method and apparatus that provide a solid state drive controller that analyzes input/output (I/O) commands from a host computing device to determine whether the commands include a vector or non-vector command, and generates a plurality of non-vector commands based on the physical addresses contained in the vector command.
- I/O input/output
- FIG. 1 is a schematic illustration of one embodiment of a system 100 including a host computing system 102 communicatively coupled to a storage device 108 .
- Storage device 108 includes a controller 110 and a non-volatile memory (NVM), such as a solid state drive (SSD) 134 made up of one or more one or more dies or planes of flash memory cells (e.g., single level cells (SLC 136 ), multi-level cells (MLC), tri level cells (TLC 138 ), etc.).
- SSD solid state drive
- Host 102 interfaces with the storage device 108 to issue I/O commands for memory operations, including: read, write, copy, and reset (e.g., erase) at the SSD 134 .
- Storage device 108 may be an internal storage drive, such as a notebook hard drive or a desktop hard drive.
- Storage device 108 may be a removable mass storage device, such as, but not limited to, a handheld, removable memory device, such as a memory card (e.g., a secure digital (SD) card, a micro secure digital (micro-SD) card, or a multimedia card (MMC)) or a universal serial bus (USB) device.
- Storage device 108 may take the form of an embedded mass storage device, such as an eSD/eMMC embedded flash drive, embedded in host 108 .
- Storage device 108 may also be any other type of internal storage device, removable storage device, embedded storage device, external storage device, or network storage device.
- Host 102 may include a wide range of devices, such as computer servers, network attached storage (NAS) units, desktop computers, notebook (i.e., laptop) computers, tablet computers (i.e., “smart” pad), set-top boxes, telephone handsets (i.e., “smart” phones), televisions, cameras, display devices, digital media players, video gaming consoles, video streaming devices, and automotive applications (i.e., mapping, autonomous driving).
- NAS network attached storage
- host 102 includes any device having a processing unit or any form of hardware capable of processing data, including a general purpose processing unit, dedicated hardware (such as an application specific integrated circuit (ASIC)), configurable hardware such as a field programmable gate array (FPGA), or any other form of processing unit configured by software instructions, microcode, or firmware.
- ASIC application specific integrated circuit
- FPGA field programmable gate array
- host 102 includes a central processing unit (CPU) 104 connected to a host memory 106 , such as DRAM or other main memories.
- An application program may be stored to host memory 106 for execution by CPU 104 .
- host memory 106 may include a computer-readable medium that stores computer-readable program code (e.g., software or firmware) executable by processor 104 .
- the host 102 can be configured with hardware and/or firmware to perform the various functions described below and shown in the flow diagrams.
- Storage device 108 includes physical (PHY) interface components 114 , for ingress of communications from host 102 to storage device 108 and egress of communications from storage device 108 to host 102 .
- a link between storage device 108 and host 102 may include any appropriate link, such as a peripheral component interconnect express (PCIe) link.
- PCIe link supports full duplex communication between storage device 108 and host 102 via dedicated unidirectional serial point-to-point connections, or “lanes.”
- PCIe link can include one to thirty-two lanes, depending on the number of available PHYs 114 between storage device 108 and host 102 .
- Communications between storage device 108 and host 102 may operate under a communication protocol, such as a PCIe serial communication protocol or other suitable communication protocols.
- a communication protocol such as a PCIe serial communication protocol or other suitable communication protocols.
- Other suitable communication protocols include ethernet, serial attached SCSI (SAS), serial AT attachment (SATA), any protocol related to remote direct memory access (RDMA) such as Infiniband, iWARP, or RDMA over Converged Ethernet (RoCE), and other suitable serial communication protocols.
- Storage device 108 may also be connected to host 102 through a switch or a bridge.
- SSD 134 of storage device 108 may include a non-volatile memory space configured for long-term storage of information after power on/off cycles.
- SSD 134 may consist of one of more dies or planes of NAND flash memory, electrically programmable read only memories (EPROM), electrically erasable programmable read only memories (EEPROM), and other solid-state memories.
- Storage device 108 includes a controller 110 which manages operations of storage device 108 , such as writes to and reads from SSD 134 .
- Controller 110 includes one or more processors 112 , which may be multi-core processors.
- Processor 112 handles the components of storage device 108 through firmware code.
- Controller 110 may operate under NVM Express (NVMe) protocol, but other protocols are applicable.
- NVMe protocol is a communications interface/protocol developed for SSDs to operate over a host and storage device linked over the PCIe interface (e.g., PHY 114 ).
- NVMe protocol provides a command submission queue and command completion queue for access of data stored in storage device 108 by host 102 .
- Storage device 108 may fetch, receive, or read host 102 commands from a command submission queue of host queues stored in host memory 106 .
- Controller 110 executes computer-readable program code (e.g., software and/or firmware) executable instructions (herein referred to as “instructions”).
- the instructions may be executed by various components of controller 110 , such as processor 112 , logic gates, switches, application specific integrated circuits (ASICs), programmable logic controllers, embedded microcontrollers, and other components of controller 110 .
- processor 112 logic gates, switches, application specific integrated circuits (ASICs), programmable logic controllers, embedded microcontrollers, and other components of controller 110 .
- ASICs application specific integrated circuits
- programmable logic controllers programmable logic controllers
- embedded microcontrollers embedded microcontrollers
- the instructions are stored in a non-transitory computer readable storage medium.
- the instructions are stored in a non-transitory computer readable storage medium of storage device 108 , such as in a read-only memory or in SSD 134 . Instructions stored in storage device 108 may be executed without added input or directions from host 102 . In other embodiments, the instructions are transmitted from host 102 .
- the controller 110 is configured with hardware and instructions to perform the various functions described herein and shown in the figures.
- Storage device 102 may also include other components, such as flash interface module 130 , a direct memory access (DMA) module 126 , a scheduler 128 , a command executor 120 , and a control path 124 .
- Flash interface module 130 interacts with SSD 134 for operations associated with various I/O commands for controlling and accessing the open-channel SSD (e.g., SSD 134 ).
- DMA module 126 is configured to control actual data transfers between host 102 and storage device 108 .
- Scheduler 128 is configured to activate and schedule the various components illustrated in FIG. 1 .
- scheduler 128 controls the data transfer while activating the control path 124 for fetching physical page regions (PRPs), posting completion requests and interrupts, and activating the DMAs for the actual data transfer between host 102 and storage device 108 .
- Control path 124 is configured to transfer control information between host 102 and storage device 108 , and provide completion requests to a vector completer 122 .
- Command parser 118 is configured to parse and queue commands received from a vector command converter 116 , and communicate the commands to command executer 120 .
- the command executer 120 is configured to arbitrate and execute the commands received from the command parser 118 .
- the vector command converter 116 and the vector completer 122 may relate to one or more of a hardware, firmware, and/or software implementation on the device controller 110 for supporting vector-type commands from host 102 .
- the vector command converter 116 monitors I/O commands received from the host 102 .
- the vector command converter 116 may determine whether the I/O command is a vector command or a non-vector command.
- the vector command converter 116 may identify the I/O command as a vector command based on whether the I/O command includes an extra field that contains a list of scatter-gather logical block addresses.
- the non-vector I/O command is communicated to the command parser for execution. Otherwise, the vector command converter 116 fetches the metadata of the vector command via the control path 124 , and temporarily stops fetching new commands from the host 102 .
- the vector command converter 116 generates a plurality of non-vector I/O commands based on a single vector command. In some cases, the entire set of the plurality of non-vector I/O commands are logically equivalent to the single vector command. In some examples, the plurality of non-vector I/O commands are generated by converting pointers contained in the vector command to physical addresses, and generating a non-vector I/O command for each physical address. The vector command converter 116 may then queue the non-vector I/O commands in the command parser 118 for execution. Once the non-vector I/O commands are queued, the device controller 110 may resume fetching new I/O commands from host 102 .
- a vector command may be a list or a set of commands generated by host 102 .
- a plurality of physical addresses may be allocated to each vector command to specify a plurality of memory locations in the SSD.
- the vector command may include a pointer to metadata stored on the host memory 106 .
- the metadata contains a physical address list for accessing SSD 134 .
- the vector completer 122 gathers all completion messages associated with the plurality of non-vector I/O commands generated by the vector command converter 116 . Initially, the gathered completion messages will not be provided to the host 102 . Instead, once all the completion messages associated with the plurality of non-vector I/O commands are received, the vector completer 122 generates a vector command completion message based on the content of all the completion requests associated with the plurality of non-vector I/O commands. That is, the vector command completion message is an indication that the vector command received from the host 102 and converted to the plurality of non-vector I/O commands has been completed. The vector completer 122 then provides the vector command completion message to the host 102 .
- device controller 110 is responsible for handling I/O commands (including vector commands) received from host 102 , ensuring data integrity and efficient storage, and managing the SSD 134 . To perform these tasks, controller 110 runs a flash translation layer (FTL) firmware. FTL tasks may be executed by one or more processors 112 utilizing one or more of the aforementioned components.
- FTL flash translation layer
- FTL may also performs numerous other functions, including: error code correction (i.e., use of redundant data, or parity data to recover a message with errors); garbage collection (i.e., identifying/invalidating stale data in SSD 134 memory blocks for deletion to free space for future writes); scheduling (i.e., controlling the order in which I/O commands from host 102 are executed); over-provisioning (i.e., memory reserved for maintaining write speed); and wear-leveling (i.e., spreading writes across the blocks of the SSD 134 as evenly as possible to ensure that all blocks of the device wear at a roughly similar rate).
- error code correction i.e., use of redundant data, or parity data to recover a message with errors
- garbage collection i.e., identifying/invalidating stale data in SSD 134 memory blocks for deletion to free space for future writes
- scheduling i.e., controlling the order in which I/O commands from host 102 are executed
- over-provisioning i.e., memory reserved for
- SSD 134 may include a physically addressable SSD, such as an open-channel SSD.
- the open-channel SSD may expose the internal parallel organization of the SSD 134 to the host 102 so that the SSD 134 can be directly managed by host 102 .
- FIG. 2 is a block diagram illustrating a logical representation of the internal parallel organization of an open-channel SSD 202 .
- the logical structure of the SSD 202 is divided by one or more groups 204 a - 204 n , parallel units 206 a - 206 n (e.g., NAND dies or planes), and chunks 208 a - 208 b containing one or more logical blocks (not shown).
- parallel units 206 a - 206 n there is a set of chunks representing a linear range of logical blocks.
- a group 204 a - 204 n may relate to a shared bus or physical channel used to communicate I/O commands to the parallel units 206 a - 206 n
- the parallel unit 206 a - 206 n may relate to one or more NAND flash memory devices on the shared bus.
- the chunk 208 a - 208 b is the smallest addressable unit for reading and writing
- a logical block is the smallest erasable unit.
- the host 102 is responsible for aspects of the FTL. Accordingly, the open-channel SSD 202 may not receive a logical address based instruction from the host 102 . Instead, host 102 may directly manage physical addresses of the open-channel SSD 202 in which data are stored. In some configurations, responsibilities of host 102 include data placement and I/O scheduling. For example, the host 102 may control data mapping using physical addressing, as well as generating mapping tables and other metadata.
- the host 102 may issue I/O commands using vector-type commands that include an extra field configured to communicate a list of scatter-gather logical block addresses.
- host 102 is able to access separate parallel units 206 a - 206 n at the same time using a single I/O command.
- not all storage devices may be capable of supporting vector commands.
- non-vector I/O commands from the host 102 are generally directed to a single NAND plane for each command, while the vector command can be directed to several NAND planes in parallel in a single I/O command.
- minimal hardware, software, and/or firmware changes can be applied to the storage device to configure the device controller 110 to support vector commands issued by the host 102 .
- FIG. 3 is a flow diagram illustrating an exemplary process 300 for receiving and executing a vector command.
- the process 300 may be performed, for example, by a storage device (e.g., such as storage device 108 of FIG. 1 ).
- the process 300 may be implemented as software and/or firmware components stored on a memory that are executed and run on one or more processors (e.g., processor 112 of FIG. 1 ).
- the transmission and reception of signals by the storage device 108 in exemplary process 300 may be enabled, for example, by one or more PHY interfaces (e.g., PHY 114 of FIG. 1 ).
- the storage device 108 receives a host I/O command.
- a device controller 110 of the storage device 108 may receive the I/O command from a host 102 via the PHY 114 interface.
- the device controller 110 includes a command register containing a number (e.g., 64, 256, 512, etc.) of 1-bit slots.
- the device controller 110 can regulate the receipt of I/O commands from the host 102 based on whether the command register has any available slots (e.g., slots with a binary “0”).
- the device controller 110 may set (e.g., with a binary “1”) one of the command register slots for each command received from the host 102 . Accordingly, if a bit is set, the corresponding slot has an I/O command received by the device controller 110 associated with it. If the bit is available, then the slot is empty and the device controller 110 is available for a new command.
- a processor 112 may determine whether the command register contains any available slots before the device controller 110 fetches a new I/O command. For example, the processor 112 may utilize a bitmap vector stored in the device controller 110 to determine whether a command register slot is set before proceeding to fetch an I/O command from the host 102 .
- the device controller 110 determines whether the command register contains at least one available slot prior to fetching an I/O command from the host 102 . Once the device controller 110 determines that a slot is available, the device controller 110 fetches the I/O command and proceeds to set the available slot. If the device controller 110 determines that there are no slots available, then the device controller 110 may disable fetching of I/O commands from the host 102 until a slot becomes available.
- the device controller 110 determines whether the fetched I/O command is a vector command. If the I/O command is not a vector command, then the process 300 moves on to block 330 , and the device controller 110 is queued to the command parser 118 for execution of the command. If the I/O command is a vector command, then the process 300 moves on to block 340 , where the device controller 110 determines the number of physical addresses contained in the vector command and sets a number of command slots equal to or less than the number of physical addresses.
- the vector command converter 116 will proceed to generate only a number of non-vector I/O commands from the vector command equal to the number of slots, while the remaining non-vector I/O commands will be generated when one or more slots become available (e.g., after completion of a command corresponding to one of the slots).
- the SSD device can hold up to 64 I/O commands from the host 102 , then 64 commands slots are implemented in the command register of the device controller 110 . If all of the 64 command slots are set, then the device controller 110 will stop fetching host commands until the completion of at least one outstanding I/O command, whereby the corresponding slot is cleared.
- the command register is 0. If the device controller 110 fetches 16 non-vector I/O commands from host 102 , then bits 0 to 15 in the command register are set to indicate that command slots 0 to 15 are occupied. However, if the device controller 110 fetches a single vector command that contains 32 physical addresses, then 32 non-vector I/O commands must be generated. At this point, device controller 110 sets 32 command slots (e.g. slots 16 to 47 in the command register). In some configurations, the device controller 110 will stop fetching new I/O commands until the non-vector commands are generated and queued for execution. In other configurations, the device controller 110 may continue fetching up to 16 more commands, based on command slot availability.
- the device controller 110 fetches 15 new non-vector commands and 1 vector command from the host 102 , and the vector command contains 21 physical addresses, then the device controller sets the remaining 16 command slots (i.e., fifteen slots are set for the non-vector commands, and 1 slot is set for the vector command), and generates only one non-vector command corresponding to one of the physical addresses in the vector command. The device controller then stops fetching I/O commands from the host 102 and waits until a slot becomes available before generating additional non-vector commands corresponding to the remaining 20 physical addresses in the vector command.
- the device controller 110 may stop fetching additional I/O commands from host 102 until a slot in the command register has been set for each physical address contained in the vector command, and the command register contains an available slot.
- the device controller 110 may continue to fetch new commands from host 102 , but it will take into account the number of I/O commands that are going to be generated. For example, the device controller 110 may continue to fetch new commands, identify the number of physical addresses in each command, and queue the vector command converter 116 to generate the plurality of I/O commands based on the vector command as slots become available.
- the device controller 110 fetches the metadata of the vector command from the host 102 .
- the metadata contains information required for the generation of the plurality of I/O commands.
- the metadata may include a list of physical addresses that correspond to a pointer included in the vector command.
- the metadata may include a pointer represented by a physical region page (PRP) or a scatter gather list (SGL), according to NVMe standard.
- PRP physical region page
- SGL scatter gather list
- the vector command converter 116 generates a plurality of non-vector I/O commands based on the vector command fetched from the host 102 .
- the vector command converter 116 generates each of the non-vector I/O commands by attaching one of the physical addresses of the vector command to each non-vector command. Accordingly, the plurality of non-vector I/O commands are equivalent to the original vector command.
- the plurality of non-vector I/O commands are generated in an order according to metadata provided by the host 102 .
- the device controller 110 may fetch the metadata from the host 102 .
- the metadata contains a physical address list. Accordingly, the non-vector I/O commands may be generated sequentially in the order of the addresses on the list.
- the non-vector I/O commands are generated according to the structure of the storage device and/or resource availability.
- the vector command converter 116 may generate the non-vector I/O commands in order of priority, where a physical address associated with an available parallel unit gets higher priority than a physical address associated with a busy parallel unit. In this configuration, quality of service and efficiency of the storage device 108 is improved.
- the vector command converter 116 sends the plurality of non-vector commands to the command parser 118 to be queued for execution.
- FIG. 4 is a flow diagram illustrating an exemplary process 400 for generating and sending a command completion.
- the process 400 may be performed, for example, by a storage device (e.g., such as storage device 108 of FIG. 1 ).
- the process 400 may be implemented as software and/or firmware components stored on a memory that are executed and run on one or more processors (e.g., processor 112 of FIG. 1 ).
- the transmission and reception of signals by the storage device 108 in exemplary process 400 may be enabled, for example, by one or more PHY interfaces (e.g., PHY 114 of FIG. 1 ).
- the vector completer 122 receives a completion request.
- the completion request is generated by the flash interface module 130 and communicated to the vector completer 122 via control path 124 .
- the vector completer 122 identifies whether the completion request corresponds to a standalone non-vector I/O command or one of a plurality of non-vector commands that originated from a vector command. If the completion request corresponds to the standalone non-vector I/O command, then the process 400 moves on to block 430 where the vector completer 122 generates a completion message and provides the completion message to the host 102 .
- the process 400 moves on to step 440 where the vector completer 122 determines whether the received completion request is the last request for the vector command.
- the vector completer 122 may be configured to gather all the completion requests associated with a particular vector command. If the received completion request is not the last request (i.e., the vector completer 122 has not gathered all the completion requests associated with each physical address of the vector command), then the process 400 moves on to block 450 , where the vector completer 122 holds the received completion request and waits until it receives the last completion request for the vector command.
- the process 400 moves on to block 460 where the vector completer 122 generates a vector command completion message based on all the received completion requests associated with the vector command.
- the vector completer 122 clears all received completion requests associated with the vector command.
- the disclosure relates generally to systems, methods, and apparatus for receiving vector-type I/O commands from host 102 , and executing the vector commands on an open-channel SSD (e.g., SSD 134 ) with minimal changes to a storage device 108 .
- certain aspects of the disclosure relate to methods and apparatus for supporting vector commands such that the vector commands are processed transparently to the host 102 and several components of the storage device 108 .
- the vector command converter 116 may be configured to generate a plurality of non-vector I/O commands from a single vector command, and pass the plurality of non-vector I/O commands onto other components of the storage device 108 for execution.
- the vector completer 122 may generate a vector command completion message based on the completion of the plurality of non-vector I/O commands, and pass the completion message to the host 102 .
- the host 102 may not be provided with an indication of how the vector command was processed.
- the generation, by the vector command converter 116 , of the plurality of non-vector I/O commands from a single vector command provides the device controller 110 with the capacity to intelligently schedule the plurality of non-vector I/O commands to improve efficiency of the storage device 108 .
- the vector command converter 116 may generate the non-vector I/O commands in order of priority, where a physical address associated with an available parallel unit gets higher priority than a physical address associated with a busy parallel unit. In this configuration, quality of service and efficiency of the storage device 108 is improved.
- a device in one non-limiting embodiment, includes a memory and a controller communicatively coupled to the memory.
- the controller is configured to identify a set of physical addresses in a vector command fetched from a host computing device, wherein each physical address of the set of physical addresses corresponds to a location in the memory.
- the controller is configured to generate a set of non-vector commands, wherein each non-vector command in the set of non-vector commands corresponds to one of each physical address in the set of physical addresses.
- the controller is configured to execute the set of non-vector commands.
- the controller is configured to generate a message to the host computing device indicating completion of the vector command in response to the execution of the set of non-vector commands.
- the controller is further configured to fetch a command from the host computing device, and determine the fetched command is a vector command, wherein the determination is based on whether the fetched command comprises a field configured to communicate a list of scatter-gather logical block addresses.
- the controller is further configured to fetch meta-data from the host computing device, wherein the meta-data comprises a list of physical addresses.
- the vector command comprises a pointer configured to identify the set of physical addresses in the list of physical addresses in the meta-data.
- the set of physical addresses comprises two or more physical addresses.
- the controller includes a command register containing a plurality of 1-bit slots, wherein the controller is further configured to determine whether the command register contains an available slot, and fetch a command from the host computing device if the command register contains the available slot.
- the controller is further configured to execute one non-vector command of the set of non-vector commands for each available slot in the command register.
- the memory comprises an open-channel solid state drive (SSD) memory device.
- SSD solid state drive
- a device in one non-limiting embodiment, includes a memory and a controller communicatively coupled to the memory.
- the controller is configured to receive an indication that the memory has executed a non-vector command, wherein the non-vector command is one of a plurality of non-vector commands corresponding to a vector command received from a host computing device.
- the controller is configured to determine whether the memory has executed all of the plurality of non-vector commands corresponding to the vector command.
- the controller is configured to generate a completion message indicating execution of the vector command if the memory has executed each of the plurality of non-vector commands. And in some configurations, the controller is configured to transmit the completion message to the host computing device.
- the controller is further configured to store the indication that the memory has executed the non-vector command if the memory has not executed all of the plurality of non-vector commands corresponding to the vector command.
- the controller is further configured to receive a plurality of indications that the memory has executed a non-vector command, wherein each of the plurality of indications correspond to one of the plurality of non-vector commands.
- the controller is configured to store the plurality of indications.
- the controller is configured to clear the stored plurality of indications after transmission of the completion message to the host computing device.
- each of the plurality of non-vector commands comprises one of a plurality of physical addresses contained in the vector command.
- each of the physical addresses corresponds to a location in the memory.
- each of the plurality of non-vector commands is configured to be executed in sequential order by the memory.
- each of the plurality of non-vector commands shares at least one of a write, read, copy, or reset command with the vector command.
- the memory comprises an open-channel non-volatile memory.
- an apparatus in one non-limiting embodiment, includes a means for receiving a vector command from a host device, wherein the vector command comprises a set of physical addresses. In some configurations, the apparatus includes a means for generating a set of non-vector commands, wherein each non-vector command in the set of non-vector commands corresponds to one of each physical address in the set of physical addresses, and wherein each non-vector command in the set of non-vector commands is configured to be executed by a means for storing digital data. In some configurations, the apparatus includes a means for generating a message to the host computing device indicating completion of the vector command in response to the execution of the set of non-vector commands by the means for storing digital data.
- the apparatus includes a means for fetching meta-data from the host computing device, wherein the meta-data comprises a list of physical addresses, and wherein the vector command comprises a pointer configured to identify the set of physical addresses in the list of physical addresses in the meta-data.
- the apparatus includes means for determining whether a command register contains an available slot, wherein the command register comprises a plurality of 1-bit slots. In some configurations, the apparatus includes a means for fetching a command from the host computing device if the command register contains the available slot.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Human Computer Interaction (AREA)
- Mathematical Physics (AREA)
- Techniques For Improving Reliability Of Storages (AREA)
- Advance Control (AREA)
Abstract
Description
- Embodiments of the present disclosure generally relate to systems and methods for executing vector commands on a memory device.
- A solid state drive (SSD) storage device is a data storage device commonly composed of a controller and an SSD memory (e.g., a non-volatile NAND flash). In all SSD storage devices, the controller sits between one or more host computing systems and the SSD memory. The SSD storage device interacts with a host using a flash-translation layer (FTL).
- Recently, a new class of SSDs have been developed known as open-channel SSDs. Open-channel SSDs differ from a traditional SSD in that they expose the internal mapping of the SSD memory structure to the host, allowing the host to perform data placement and data input/output (I/O) scheduling using the physical addresses instead of the traditional logical addresses. Accordingly, the open-channel SSD has the ability to leave certain aspects of the FTL for the host to handle. As the host takes on additional responsibilities that were once relegated to the storage device, additional commands, such as vector commands may be used to enhance the functionality of the SSDs.
- Thus, as the demand for digital storage solutions continue to increase, what is needed are new ways to support the additional vector commands not only to meet the growing demand, but to advance and enhance the user experience.
- The present disclosure generally relates to a method and apparatus that provide a solid state drive controller that analyzes input/output commands from a host computing device to determine whether the commands include a vector or non-vector command, and is configured to generate a plurality of non-vector commands based on the physical addresses contained in the vector command.
- In one embodiment, the disclosure describes a device, comprising a memory and a controller communicatively coupled to the memory, wherein the controller is configured to identify a set of physical addresses in a vector command fetched from a host computing device, wherein each physical address of the set of physical addresses corresponds to a location in the memory. In some embodiments, the controller is also configured to generate a set of non-vector commands, wherein each non-vector command in the set of non-vector commands corresponds to one of each physical address in the set of physical addresses. In some embodiments, the controller is also configured to execute the set of non-vector commands. In some embodiments, the controller is configured to generate a message to the host computing device indicating completion of the vector command in response to the execution of the set of non-vector commands.
- In another embodiment, the disclosure describes device, comprising a memory and a controller communicatively coupled to the memory, wherein the controller is configured to receive an indication that the memory has executed a non-vector command, wherein the non-vector command is one of a plurality of non-vector commands corresponding to a vector command received from a host computing device. In some embodiments, the controller is also configured to determine whether the memory has executed all of the plurality of non-vector commands corresponding to the vector command. In some embodiments, the controller is also configured to generate a completion message indicating execution of the vector command if the memory has executed each of the plurality of non-vector commands. In some embodiments, the controller is also configured to transmit the completion message to the host computing device.
- In another embodiment, the disclosure describes an apparatus. In one example, the apparatus includes means for receiving a vector command from a host device, wherein the vector command comprises a set of physical addresses. In another example, the apparatus include a means for generating a set of non-vector commands, wherein each non-vector command in the set of non-vector commands corresponds to one of each physical address in the set of physical addresses, and wherein each non-vector command in the set of non-vector commands is configured to be executed by a means for storing digital data. In another example, the apparatus includes a means for generating a message to the host computing device indicating completion of the vector command in response to the execution of the set of non-vector commands by the means for storing digital data.
- So that the manner in which the above recited features of the present disclosure can be understood in detail, a more particular description of the disclosure, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments of this disclosure and are therefore not to be considered limiting of its scope, for the disclosure may admit to other equally effective embodiments.
-
FIG. 1 is a block diagram of a memory device communicatively coupled to a host computer system, in accordance with certain aspects of the disclosure. -
FIG. 2 is a block diagram illustrating a logical representation of the internal parallel organization of an open-channel solid state drive (SSD), in accordance with certain aspects of the disclosure. -
FIG. 3 is a flow diagram illustrating an exemplary process for executing a vector command in accordance with certain aspects of the disclosure. -
FIG. 4 is a flow diagram illustrating an exemplary process for generating a vector command completion message in accordance with certain aspects of the disclosure. - To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements disclosed in one embodiment may be beneficially utilized on other embodiments without specific recitation.
- In the following, reference is made to embodiments of the disclosure. However, it should be understood that the disclosure is not limited to specific described embodiments. Instead, any combination of the following features and elements, whether related to different embodiments or not, is contemplated to implement and practice the disclosure. Furthermore, although embodiments of the disclosure may achieve advantages over other possible solutions and/or over the prior art, whether or not a particular advantage is achieved by a given embodiment is not limiting of the disclosure. Thus, the following aspects, features, embodiments and advantages are merely illustrative and are not considered elements or limitations of the appended claims except where explicitly recited in a claim(s). Likewise, reference to “the disclosure” shall not be construed as a generalization of any inventive subject matter disclosed herein and shall not be considered to be an element or limitation of the appended claims except where explicitly recited in a claim(s).
- The present disclosure relates to a method and apparatus that provide a solid state drive controller that analyzes input/output (I/O) commands from a host computing device to determine whether the commands include a vector or non-vector command, and generates a plurality of non-vector commands based on the physical addresses contained in the vector command.
-
FIG. 1 is a schematic illustration of one embodiment of asystem 100 including ahost computing system 102 communicatively coupled to astorage device 108.Storage device 108 includes acontroller 110 and a non-volatile memory (NVM), such as a solid state drive (SSD) 134 made up of one or more one or more dies or planes of flash memory cells (e.g., single level cells (SLC 136), multi-level cells (MLC), tri level cells (TLC 138), etc.). Host 102 interfaces with thestorage device 108 to issue I/O commands for memory operations, including: read, write, copy, and reset (e.g., erase) at the SSD 134. -
Storage device 108 may be an internal storage drive, such as a notebook hard drive or a desktop hard drive.Storage device 108 may be a removable mass storage device, such as, but not limited to, a handheld, removable memory device, such as a memory card (e.g., a secure digital (SD) card, a micro secure digital (micro-SD) card, or a multimedia card (MMC)) or a universal serial bus (USB) device.Storage device 108 may take the form of an embedded mass storage device, such as an eSD/eMMC embedded flash drive, embedded inhost 108.Storage device 108 may also be any other type of internal storage device, removable storage device, embedded storage device, external storage device, or network storage device. -
Host 102 may include a wide range of devices, such as computer servers, network attached storage (NAS) units, desktop computers, notebook (i.e., laptop) computers, tablet computers (i.e., “smart” pad), set-top boxes, telephone handsets (i.e., “smart” phones), televisions, cameras, display devices, digital media players, video gaming consoles, video streaming devices, and automotive applications (i.e., mapping, autonomous driving). In certain embodiments,host 102 includes any device having a processing unit or any form of hardware capable of processing data, including a general purpose processing unit, dedicated hardware (such as an application specific integrated circuit (ASIC)), configurable hardware such as a field programmable gate array (FPGA), or any other form of processing unit configured by software instructions, microcode, or firmware. - In some configurations,
host 102 includes a central processing unit (CPU) 104 connected to ahost memory 106, such as DRAM or other main memories. An application program may be stored to hostmemory 106 for execution byCPU 104. For example,host memory 106 may include a computer-readable medium that stores computer-readable program code (e.g., software or firmware) executable byprocessor 104. Thehost 102 can be configured with hardware and/or firmware to perform the various functions described below and shown in the flow diagrams. -
Storage device 108 includes physical (PHY)interface components 114, for ingress of communications fromhost 102 tostorage device 108 and egress of communications fromstorage device 108 tohost 102. A link betweenstorage device 108 andhost 102 may include any appropriate link, such as a peripheral component interconnect express (PCIe) link. PCIe link supports full duplex communication betweenstorage device 108 andhost 102 via dedicated unidirectional serial point-to-point connections, or “lanes.” PCIe link can include one to thirty-two lanes, depending on the number ofavailable PHYs 114 betweenstorage device 108 andhost 102. - Communications between
storage device 108 andhost 102 may operate under a communication protocol, such as a PCIe serial communication protocol or other suitable communication protocols. Other suitable communication protocols include ethernet, serial attached SCSI (SAS), serial AT attachment (SATA), any protocol related to remote direct memory access (RDMA) such as Infiniband, iWARP, or RDMA over Converged Ethernet (RoCE), and other suitable serial communication protocols.Storage device 108 may also be connected tohost 102 through a switch or a bridge. - SSD 134 of
storage device 108 may include a non-volatile memory space configured for long-term storage of information after power on/off cycles. In some examples, SSD 134 may consist of one of more dies or planes of NAND flash memory, electrically programmable read only memories (EPROM), electrically erasable programmable read only memories (EEPROM), and other solid-state memories. -
Storage device 108 includes acontroller 110 which manages operations ofstorage device 108, such as writes to and reads from SSD 134.Controller 110 includes one ormore processors 112, which may be multi-core processors.Processor 112 handles the components ofstorage device 108 through firmware code. -
Controller 110 may operate under NVM Express (NVMe) protocol, but other protocols are applicable. NVMe protocol is a communications interface/protocol developed for SSDs to operate over a host and storage device linked over the PCIe interface (e.g., PHY 114). NVMe protocol provides a command submission queue and command completion queue for access of data stored instorage device 108 byhost 102.Storage device 108 may fetch, receive, or readhost 102 commands from a command submission queue of host queues stored inhost memory 106. -
Controller 110 executes computer-readable program code (e.g., software and/or firmware) executable instructions (herein referred to as “instructions”). The instructions may be executed by various components ofcontroller 110, such asprocessor 112, logic gates, switches, application specific integrated circuits (ASICs), programmable logic controllers, embedded microcontrollers, and other components ofcontroller 110. - The instructions are stored in a non-transitory computer readable storage medium. In some embodiments, the instructions are stored in a non-transitory computer readable storage medium of
storage device 108, such as in a read-only memory or inSSD 134. Instructions stored instorage device 108 may be executed without added input or directions fromhost 102. In other embodiments, the instructions are transmitted fromhost 102. Thecontroller 110 is configured with hardware and instructions to perform the various functions described herein and shown in the figures. -
Storage device 102 may also include other components, such asflash interface module 130, a direct memory access (DMA)module 126, ascheduler 128, acommand executor 120, and acontrol path 124.Flash interface module 130 interacts withSSD 134 for operations associated with various I/O commands for controlling and accessing the open-channel SSD (e.g., SSD 134).DMA module 126 is configured to control actual data transfers betweenhost 102 andstorage device 108.Scheduler 128 is configured to activate and schedule the various components illustrated inFIG. 1 . For example,scheduler 128 controls the data transfer while activating thecontrol path 124 for fetching physical page regions (PRPs), posting completion requests and interrupts, and activating the DMAs for the actual data transfer betweenhost 102 andstorage device 108.Control path 124 is configured to transfer control information betweenhost 102 andstorage device 108, and provide completion requests to avector completer 122.Command parser 118 is configured to parse and queue commands received from avector command converter 116, and communicate the commands to commandexecuter 120. Thecommand executer 120 is configured to arbitrate and execute the commands received from thecommand parser 118. - The
vector command converter 116 and the vector completer 122 may relate to one or more of a hardware, firmware, and/or software implementation on thedevice controller 110 for supporting vector-type commands fromhost 102. In some configurations, thevector command converter 116 monitors I/O commands received from thehost 102. Thevector command converter 116 may determine whether the I/O command is a vector command or a non-vector command. For example, thevector command converter 116 may identify the I/O command as a vector command based on whether the I/O command includes an extra field that contains a list of scatter-gather logical block addresses. If the I/O command is not a vector command (e.g., does not contain the list of scatter-gather logical block addresses), the non-vector I/O command is communicated to the command parser for execution. Otherwise, thevector command converter 116 fetches the metadata of the vector command via thecontrol path 124, and temporarily stops fetching new commands from thehost 102. - In some configurations, the
vector command converter 116 generates a plurality of non-vector I/O commands based on a single vector command. In some cases, the entire set of the plurality of non-vector I/O commands are logically equivalent to the single vector command. In some examples, the plurality of non-vector I/O commands are generated by converting pointers contained in the vector command to physical addresses, and generating a non-vector I/O command for each physical address. Thevector command converter 116 may then queue the non-vector I/O commands in thecommand parser 118 for execution. Once the non-vector I/O commands are queued, thedevice controller 110 may resume fetching new I/O commands fromhost 102. As used herein, a vector command may be a list or a set of commands generated byhost 102. A plurality of physical addresses may be allocated to each vector command to specify a plurality of memory locations in the SSD. In some configurations, the vector command may include a pointer to metadata stored on thehost memory 106. In this configuration, the metadata contains a physical address list for accessingSSD 134. - In some configurations, the
vector completer 122 gathers all completion messages associated with the plurality of non-vector I/O commands generated by thevector command converter 116. Initially, the gathered completion messages will not be provided to thehost 102. Instead, once all the completion messages associated with the plurality of non-vector I/O commands are received, thevector completer 122 generates a vector command completion message based on the content of all the completion requests associated with the plurality of non-vector I/O commands. That is, the vector command completion message is an indication that the vector command received from thehost 102 and converted to the plurality of non-vector I/O commands has been completed. The vector completer 122 then provides the vector command completion message to thehost 102. - In some configurations,
device controller 110 is responsible for handling I/O commands (including vector commands) received fromhost 102, ensuring data integrity and efficient storage, and managing theSSD 134. To perform these tasks,controller 110 runs a flash translation layer (FTL) firmware. FTL tasks may be executed by one ormore processors 112 utilizing one or more of the aforementioned components. FTL may also performs numerous other functions, including: error code correction (i.e., use of redundant data, or parity data to recover a message with errors); garbage collection (i.e., identifying/invalidating stale data inSSD 134 memory blocks for deletion to free space for future writes); scheduling (i.e., controlling the order in which I/O commands fromhost 102 are executed); over-provisioning (i.e., memory reserved for maintaining write speed); and wear-leveling (i.e., spreading writes across the blocks of theSSD 134 as evenly as possible to ensure that all blocks of the device wear at a roughly similar rate). - However, in some configurations, host 102 may be responsible for some or all of the FTL tasks. For example,
SSD 134 may include a physically addressable SSD, such as an open-channel SSD. In such a configuration, the open-channel SSD may expose the internal parallel organization of theSSD 134 to thehost 102 so that theSSD 134 can be directly managed byhost 102. -
FIG. 2 is a block diagram illustrating a logical representation of the internal parallel organization of an open-channel SSD 202. The logical structure of the SSD 202 is divided by one or more groups 204 a-204 n, parallel units 206 a-206 n (e.g., NAND dies or planes), and chunks 208 a-208 b containing one or more logical blocks (not shown). Within the parallel units 206 a-206 n, there is a set of chunks representing a linear range of logical blocks. In some examples, a group 204 a-204 n may relate to a shared bus or physical channel used to communicate I/O commands to the parallel units 206 a-206 n, while the parallel unit 206 a-206 n may relate to one or more NAND flash memory devices on the shared bus. In some examples, the chunk 208 a-208 b is the smallest addressable unit for reading and writing, and a logical block is the smallest erasable unit. - In some configurations, the
host 102 is responsible for aspects of the FTL. Accordingly, the open-channel SSD 202 may not receive a logical address based instruction from thehost 102. Instead, host 102 may directly manage physical addresses of the open-channel SSD 202 in which data are stored. In some configurations, responsibilities ofhost 102 include data placement and I/O scheduling. For example, thehost 102 may control data mapping using physical addressing, as well as generating mapping tables and other metadata. - Accordingly, the
host 102 may issue I/O commands using vector-type commands that include an extra field configured to communicate a list of scatter-gather logical block addresses. Using the vector command,host 102 is able to access separate parallel units 206 a-206 n at the same time using a single I/O command. However, not all storage devices may be capable of supporting vector commands. For example, non-vector I/O commands from thehost 102 are generally directed to a single NAND plane for each command, while the vector command can be directed to several NAND planes in parallel in a single I/O command. According to certain aspects of the disclosure, minimal hardware, software, and/or firmware changes can be applied to the storage device to configure thedevice controller 110 to support vector commands issued by thehost 102. -
FIG. 3 is a flow diagram illustrating anexemplary process 300 for receiving and executing a vector command. Theprocess 300 may be performed, for example, by a storage device (e.g., such asstorage device 108 ofFIG. 1 ). Theprocess 300 may be implemented as software and/or firmware components stored on a memory that are executed and run on one or more processors (e.g.,processor 112 ofFIG. 1 ). Further, the transmission and reception of signals by thestorage device 108 inexemplary process 300 may be enabled, for example, by one or more PHY interfaces (e.g.,PHY 114 ofFIG. 1 ). - At
block 310, thestorage device 108 receives a host I/O command. For example, adevice controller 110 of thestorage device 108 may receive the I/O command from ahost 102 via thePHY 114 interface. In some configurations, thedevice controller 110 includes a command register containing a number (e.g., 64, 256, 512, etc.) of 1-bit slots. Thedevice controller 110 can regulate the receipt of I/O commands from thehost 102 based on whether the command register has any available slots (e.g., slots with a binary “0”). - For example, the
device controller 110 may set (e.g., with a binary “1”) one of the command register slots for each command received from thehost 102. Accordingly, if a bit is set, the corresponding slot has an I/O command received by thedevice controller 110 associated with it. If the bit is available, then the slot is empty and thedevice controller 110 is available for a new command. - In some configurations, a
processor 112 may determine whether the command register contains any available slots before thedevice controller 110 fetches a new I/O command. For example, theprocessor 112 may utilize a bitmap vector stored in thedevice controller 110 to determine whether a command register slot is set before proceeding to fetch an I/O command from thehost 102. - Accordingly, in some configurations, the
device controller 110 determines whether the command register contains at least one available slot prior to fetching an I/O command from thehost 102. Once thedevice controller 110 determines that a slot is available, thedevice controller 110 fetches the I/O command and proceeds to set the available slot. If thedevice controller 110 determines that there are no slots available, then thedevice controller 110 may disable fetching of I/O commands from thehost 102 until a slot becomes available. - At
block 320, thedevice controller 110 determines whether the fetched I/O command is a vector command. If the I/O command is not a vector command, then theprocess 300 moves on to block 330, and thedevice controller 110 is queued to thecommand parser 118 for execution of the command. If the I/O command is a vector command, then theprocess 300 moves on to block 340, where thedevice controller 110 determines the number of physical addresses contained in the vector command and sets a number of command slots equal to or less than the number of physical addresses. For example, if there are fewer available slots in the command register than physical addresses contained in the vector command, then thevector command converter 116 will proceed to generate only a number of non-vector I/O commands from the vector command equal to the number of slots, while the remaining non-vector I/O commands will be generated when one or more slots become available (e.g., after completion of a command corresponding to one of the slots). - For example, if the SSD device can hold up to 64 I/O commands from the
host 102, then 64 commands slots are implemented in the command register of thedevice controller 110. If all of the 64 command slots are set, then thedevice controller 110 will stop fetching host commands until the completion of at least one outstanding I/O command, whereby the corresponding slot is cleared. - Similarly, if the
storage device 108 is in an idle state, and there are no outstanding commands in thedevice controller 110, then the command register is 0. If thedevice controller 110 fetches 16 non-vector I/O commands fromhost 102, then bits 0 to 15 in the command register are set to indicate that command slots 0 to 15 are occupied. However, if thedevice controller 110 fetches a single vector command that contains 32 physical addresses, then 32 non-vector I/O commands must be generated. At this point,device controller 110 sets 32 command slots (e.g. slots 16 to 47 in the command register). In some configurations, thedevice controller 110 will stop fetching new I/O commands until the non-vector commands are generated and queued for execution. In other configurations, thedevice controller 110 may continue fetching up to 16 more commands, based on command slot availability. - For example, if the
device controller 110 fetches 15 new non-vector commands and 1 vector command from thehost 102, and the vector command contains 21 physical addresses, then the device controller sets the remaining 16 command slots (i.e., fifteen slots are set for the non-vector commands, and 1 slot is set for the vector command), and generates only one non-vector command corresponding to one of the physical addresses in the vector command. The device controller then stops fetching I/O commands from thehost 102 and waits until a slot becomes available before generating additional non-vector commands corresponding to the remaining 20 physical addresses in the vector command. - Accordingly, the
device controller 110 may stop fetching additional I/O commands fromhost 102 until a slot in the command register has been set for each physical address contained in the vector command, and the command register contains an available slot. In other configurations, if the command register does not contain an available slot, thedevice controller 110 may continue to fetch new commands fromhost 102, but it will take into account the number of I/O commands that are going to be generated. For example, thedevice controller 110 may continue to fetch new commands, identify the number of physical addresses in each command, and queue thevector command converter 116 to generate the plurality of I/O commands based on the vector command as slots become available. - At
block 350, thedevice controller 110 fetches the metadata of the vector command from thehost 102. In some configurations, the metadata contains information required for the generation of the plurality of I/O commands. For example, the metadata may include a list of physical addresses that correspond to a pointer included in the vector command. In another example, the metadata may include a pointer represented by a physical region page (PRP) or a scatter gather list (SGL), according to NVMe standard. Storage device fetches the pointers and may manipulate them in order to build the pointers for each non-vector command. - At
block 360, thevector command converter 116 generates a plurality of non-vector I/O commands based on the vector command fetched from thehost 102. In some configurations, thevector command converter 116 generates each of the non-vector I/O commands by attaching one of the physical addresses of the vector command to each non-vector command. Accordingly, the plurality of non-vector I/O commands are equivalent to the original vector command. - In some configurations, the plurality of non-vector I/O commands are generated in an order according to metadata provided by the
host 102. For example, thedevice controller 110 may fetch the metadata from thehost 102. In some examples, the metadata contains a physical address list. Accordingly, the non-vector I/O commands may be generated sequentially in the order of the addresses on the list. In other configurations, the non-vector I/O commands are generated according to the structure of the storage device and/or resource availability. For example, thevector command converter 116 may generate the non-vector I/O commands in order of priority, where a physical address associated with an available parallel unit gets higher priority than a physical address associated with a busy parallel unit. In this configuration, quality of service and efficiency of thestorage device 108 is improved. - At
block 370, thevector command converter 116 sends the plurality of non-vector commands to thecommand parser 118 to be queued for execution. -
FIG. 4 is a flow diagram illustrating anexemplary process 400 for generating and sending a command completion. Theprocess 400 may be performed, for example, by a storage device (e.g., such asstorage device 108 ofFIG. 1 ). Theprocess 400 may be implemented as software and/or firmware components stored on a memory that are executed and run on one or more processors (e.g.,processor 112 ofFIG. 1 ). Further, the transmission and reception of signals by thestorage device 108 inexemplary process 400 may be enabled, for example, by one or more PHY interfaces (e.g.,PHY 114 ofFIG. 1 ). - At
block 410, thevector completer 122 receives a completion request. In some configurations, the completion request is generated by theflash interface module 130 and communicated to the vector completer 122 viacontrol path 124. - At
step 420, thevector completer 122 identifies whether the completion request corresponds to a standalone non-vector I/O command or one of a plurality of non-vector commands that originated from a vector command. If the completion request corresponds to the standalone non-vector I/O command, then theprocess 400 moves on to block 430 where thevector completer 122 generates a completion message and provides the completion message to thehost 102. - If, at
step 420, thevector completer 122 determines that the completion request corresponds to one of the plurality of non-vector commands that originated from a vector command, then theprocess 400 moves on to step 440 where thevector completer 122 determines whether the received completion request is the last request for the vector command. For example, the vector completer 122 may be configured to gather all the completion requests associated with a particular vector command. If the received completion request is not the last request (i.e., thevector completer 122 has not gathered all the completion requests associated with each physical address of the vector command), then theprocess 400 moves on to block 450, where thevector completer 122 holds the received completion request and waits until it receives the last completion request for the vector command. - If, at
step 440, thevector completer 122 determines that the received completion request is the last request for the vector command, then theprocess 400 moves on to block 460 where thevector completer 122 generates a vector command completion message based on all the received completion requests associated with the vector command. - At
block 470, thevector completer 122 clears all received completion requests associated with the vector command. - In summary, the disclosure relates generally to systems, methods, and apparatus for receiving vector-type I/O commands from
host 102, and executing the vector commands on an open-channel SSD (e.g., SSD 134) with minimal changes to astorage device 108. In particular, certain aspects of the disclosure relate to methods and apparatus for supporting vector commands such that the vector commands are processed transparently to thehost 102 and several components of thestorage device 108. For example, thevector command converter 116 may be configured to generate a plurality of non-vector I/O commands from a single vector command, and pass the plurality of non-vector I/O commands onto other components of thestorage device 108 for execution. Moreover, the vector completer 122 may generate a vector command completion message based on the completion of the plurality of non-vector I/O commands, and pass the completion message to thehost 102. Here, thehost 102 may not be provided with an indication of how the vector command was processed. - In some examples, the generation, by the
vector command converter 116, of the plurality of non-vector I/O commands from a single vector command provides thedevice controller 110 with the capacity to intelligently schedule the plurality of non-vector I/O commands to improve efficiency of thestorage device 108. In one example, thevector command converter 116 may generate the non-vector I/O commands in order of priority, where a physical address associated with an available parallel unit gets higher priority than a physical address associated with a busy parallel unit. In this configuration, quality of service and efficiency of thestorage device 108 is improved. - In one non-limiting embodiment, a device is disclosed, wherein the device includes a memory and a controller communicatively coupled to the memory. In some configurations, the controller is configured to identify a set of physical addresses in a vector command fetched from a host computing device, wherein each physical address of the set of physical addresses corresponds to a location in the memory. In some configurations, the controller is configured to generate a set of non-vector commands, wherein each non-vector command in the set of non-vector commands corresponds to one of each physical address in the set of physical addresses. In some configurations, the controller is configured to execute the set of non-vector commands. In some configurations, the controller is configured to generate a message to the host computing device indicating completion of the vector command in response to the execution of the set of non-vector commands.
- In one non-limiting embodiment, the controller is further configured to fetch a command from the host computing device, and determine the fetched command is a vector command, wherein the determination is based on whether the fetched command comprises a field configured to communicate a list of scatter-gather logical block addresses.
- In one non-limiting embodiment, the controller is further configured to fetch meta-data from the host computing device, wherein the meta-data comprises a list of physical addresses.
- In one non-limiting embodiment, the vector command comprises a pointer configured to identify the set of physical addresses in the list of physical addresses in the meta-data.
- In one non-limiting embodiment, the set of physical addresses comprises two or more physical addresses.
- In one non-limiting embodiment, the controller includes a command register containing a plurality of 1-bit slots, wherein the controller is further configured to determine whether the command register contains an available slot, and fetch a command from the host computing device if the command register contains the available slot.
- In one non-limiting embodiment, the controller is further configured to execute one non-vector command of the set of non-vector commands for each available slot in the command register.
- In one non-limiting embodiment, the memory comprises an open-channel solid state drive (SSD) memory device.
- In one non-limiting embodiment, a device is disclosed, wherein the device includes a memory and a controller communicatively coupled to the memory. In some configurations, the controller is configured to receive an indication that the memory has executed a non-vector command, wherein the non-vector command is one of a plurality of non-vector commands corresponding to a vector command received from a host computing device. In some configurations, the controller is configured to determine whether the memory has executed all of the plurality of non-vector commands corresponding to the vector command. In some configurations, the controller is configured to generate a completion message indicating execution of the vector command if the memory has executed each of the plurality of non-vector commands. And in some configurations, the controller is configured to transmit the completion message to the host computing device.
- In one non-limiting embodiment, the controller is further configured to store the indication that the memory has executed the non-vector command if the memory has not executed all of the plurality of non-vector commands corresponding to the vector command.
- In one non-limiting embodiment, the controller is further configured to receive a plurality of indications that the memory has executed a non-vector command, wherein each of the plurality of indications correspond to one of the plurality of non-vector commands. In some configurations, the controller is configured to store the plurality of indications. In some configurations, the controller is configured to clear the stored plurality of indications after transmission of the completion message to the host computing device.
- In one non-limiting embodiment, each of the plurality of non-vector commands comprises one of a plurality of physical addresses contained in the vector command.
- In one non-limiting embodiment, each of the physical addresses corresponds to a location in the memory.
- In one non-limiting embodiment, each of the plurality of non-vector commands is configured to be executed in sequential order by the memory.
- In one non-limiting embodiment, each of the plurality of non-vector commands shares at least one of a write, read, copy, or reset command with the vector command.
- In one non-limiting embodiment, the memory comprises an open-channel non-volatile memory.
- In one non-limiting embodiment, an apparatus is disclosed. In some configurations, the apparatus includes a means for receiving a vector command from a host device, wherein the vector command comprises a set of physical addresses. In some configurations, the apparatus includes a means for generating a set of non-vector commands, wherein each non-vector command in the set of non-vector commands corresponds to one of each physical address in the set of physical addresses, and wherein each non-vector command in the set of non-vector commands is configured to be executed by a means for storing digital data. In some configurations, the apparatus includes a means for generating a message to the host computing device indicating completion of the vector command in response to the execution of the set of non-vector commands by the means for storing digital data.
- In one non-limiting embodiment, the apparatus includes a means for fetching meta-data from the host computing device, wherein the meta-data comprises a list of physical addresses, and wherein the vector command comprises a pointer configured to identify the set of physical addresses in the list of physical addresses in the meta-data.
- In one non-limiting embodiment, the apparatus includes means for determining whether a command register contains an available slot, wherein the command register comprises a plurality of 1-bit slots. In some configurations, the apparatus includes a means for fetching a command from the host computing device if the command register contains the available slot.
- While the foregoing is directed to embodiments of the present disclosure, other and further embodiments of the disclosure may be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow.
Claims (20)
Priority Applications (6)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/459,183 US10901733B1 (en) | 2019-07-01 | 2019-07-01 | Open channel vector command execution |
US16/739,095 US10901624B1 (en) | 2019-07-01 | 2020-01-09 | Dummy host command generation for supporting higher maximum data transfer sizes (MDTS) |
KR1020217018293A KR102645983B1 (en) | 2019-07-01 | 2020-03-17 | Open channel vector command execution |
DE112020000124.5T DE112020000124T5 (en) | 2019-07-01 | 2020-03-17 | EXECUTION OF OPEN CHANNEL VECTOR COMMANDS |
PCT/US2020/023177 WO2021002896A1 (en) | 2019-07-01 | 2020-03-17 | Open channel vector command execution |
CN202080007005.0A CN113196225A (en) | 2019-07-01 | 2020-03-17 | Open channel vector command execution |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/459,183 US10901733B1 (en) | 2019-07-01 | 2019-07-01 | Open channel vector command execution |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/739,095 Continuation-In-Part US10901624B1 (en) | 2019-07-01 | 2020-01-09 | Dummy host command generation for supporting higher maximum data transfer sizes (MDTS) |
Publications (2)
Publication Number | Publication Date |
---|---|
US20210004229A1 true US20210004229A1 (en) | 2021-01-07 |
US10901733B1 US10901733B1 (en) | 2021-01-26 |
Family
ID=74065216
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/459,183 Active 2039-08-04 US10901733B1 (en) | 2019-07-01 | 2019-07-01 | Open channel vector command execution |
Country Status (5)
Country | Link |
---|---|
US (1) | US10901733B1 (en) |
KR (1) | KR102645983B1 (en) |
CN (1) | CN113196225A (en) |
DE (1) | DE112020000124T5 (en) |
WO (1) | WO2021002896A1 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20220253246A1 (en) * | 2021-02-08 | 2022-08-11 | Samsung Electronics Co., Ltd. | Memory controller and memory control method |
US12001270B2 (en) * | 2021-12-07 | 2024-06-04 | Microchip Technology Incorporated | Vector fetch bus error handling |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11934684B2 (en) | 2021-12-14 | 2024-03-19 | Western Digital Technologies, Inc. | Maximum data transfer size per tenant and command type |
Family Cites Families (23)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP3205989B2 (en) * | 1988-09-22 | 2001-09-04 | 日本電気株式会社 | Information processing system |
US7627735B2 (en) * | 2005-10-21 | 2009-12-01 | Intel Corporation | Implementing vector memory operations |
US20070195367A1 (en) | 2006-02-22 | 2007-08-23 | Seiko Epson Corporation | Computer readable medium recording a printer control program, printer control method, and printer system |
US7849294B2 (en) * | 2008-01-31 | 2010-12-07 | International Business Machines Corporation | Sharing data in internal and memory representations with dynamic data-driven conversion |
US8407407B1 (en) | 2009-10-06 | 2013-03-26 | Marvell International Ltd. | Solid state drive access control system with equalized access timing |
US9075708B1 (en) * | 2011-06-30 | 2015-07-07 | Western Digital Technologies, Inc. | System and method for improving data integrity and power-on performance in storage devices |
US9652376B2 (en) | 2013-01-28 | 2017-05-16 | Radian Memory Systems, Inc. | Cooperative flash memory control |
US9268541B2 (en) * | 2013-03-15 | 2016-02-23 | Intel Corporation | Methods and systems to vectorize scalar computer program loops having loop-carried dependences |
US9720860B2 (en) | 2014-06-06 | 2017-08-01 | Toshiba Corporation | System and method for efficient processing of queued read commands in a memory system |
US9645739B2 (en) | 2014-09-26 | 2017-05-09 | Intel Corporation | Host-managed non-volatile memory |
US9514058B2 (en) * | 2014-12-22 | 2016-12-06 | Texas Instruments Incorporated | Local page translation and permissions storage for the page window in program memory controller |
US9904609B2 (en) | 2015-11-04 | 2018-02-27 | Toshiba Memory Corporation | Memory controller and memory device |
KR102564165B1 (en) | 2016-04-25 | 2023-08-04 | 삼성전자주식회사 | Method of managing input/output (i/o) queues by non volatile memory express (nvme) controller |
CN107870727B (en) | 2016-09-23 | 2021-01-01 | 伊姆西Ip控股有限责任公司 | Method and apparatus for storing data |
KR102618699B1 (en) * | 2016-09-28 | 2024-01-02 | 삼성전자주식회사 | Computing system including storage device controlled by host |
KR102387922B1 (en) | 2017-02-07 | 2022-04-15 | 삼성전자주식회사 | Methods and systems for handling asynchronous event request command in a solid state drive |
US10073640B1 (en) | 2017-03-10 | 2018-09-11 | Toshiba Memory Corporation | Large scale implementation of a plurality of open channel solid state drives |
KR102293069B1 (en) | 2017-09-08 | 2021-08-27 | 삼성전자주식회사 | Storage device including nonvolatile memory device and controller, controller and operating method of nonvolatile memory device |
KR102340287B1 (en) * | 2017-09-20 | 2021-12-20 | 삼성전자주식회사 | Storage device communicating with host according to multicast communication protocol and communication method of host |
CN109542335B (en) | 2017-09-22 | 2022-04-01 | 慧荣科技股份有限公司 | Data internal moving method of flash memory and device using the same |
KR102446733B1 (en) | 2017-11-30 | 2022-09-23 | 삼성전자주식회사 | Storage device and electronic device including the same |
US10761775B2 (en) | 2018-02-09 | 2020-09-01 | Samsung Electronics Co., Ltd. | System and method for NVMe inter command association in SSD storage using a bridge device |
JP7131053B2 (en) * | 2018-04-24 | 2022-09-06 | 富士通株式会社 | Storage device, information processing program and information processing system |
-
2019
- 2019-07-01 US US16/459,183 patent/US10901733B1/en active Active
-
2020
- 2020-03-17 CN CN202080007005.0A patent/CN113196225A/en active Pending
- 2020-03-17 DE DE112020000124.5T patent/DE112020000124T5/en active Pending
- 2020-03-17 WO PCT/US2020/023177 patent/WO2021002896A1/en active Application Filing
- 2020-03-17 KR KR1020217018293A patent/KR102645983B1/en active IP Right Grant
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20220253246A1 (en) * | 2021-02-08 | 2022-08-11 | Samsung Electronics Co., Ltd. | Memory controller and memory control method |
US11893278B2 (en) * | 2021-02-08 | 2024-02-06 | Samsung Electronics Co., Ltd. | Memory controller and memory control method for generating commands based on a memory request |
US20240143230A1 (en) * | 2021-02-08 | 2024-05-02 | Samsung Electronics Co., Ltd. | Memory controller and memory control method |
US12001270B2 (en) * | 2021-12-07 | 2024-06-04 | Microchip Technology Incorporated | Vector fetch bus error handling |
Also Published As
Publication number | Publication date |
---|---|
KR102645983B1 (en) | 2024-03-08 |
CN113196225A (en) | 2021-07-30 |
KR20210084632A (en) | 2021-07-07 |
US10901733B1 (en) | 2021-01-26 |
WO2021002896A1 (en) | 2021-01-07 |
DE112020000124T5 (en) | 2021-07-29 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11960725B2 (en) | NVMe controller memory manager providing CMB capability | |
US10901624B1 (en) | Dummy host command generation for supporting higher maximum data transfer sizes (MDTS) | |
US10732893B2 (en) | Non-volatile memory over fabric controller with memory bypass | |
US11481118B2 (en) | Storage media programming with adaptive write buffer release | |
US9489141B2 (en) | Efficient scheduling of Input/Output requests to reduce latency and maximize throughput in a flash storage device | |
US11188251B2 (en) | Partitioned non-volatile memory express protocol for controller memory buffer | |
CN108153482B (en) | IO command processing method and medium interface controller | |
TWI463314B (en) | Apparatus including memory system controllers and related methods | |
US10558367B2 (en) | Adaptive transaction layer packet for latency balancing | |
CN111427808A (en) | System and method for managing communication between a storage device and a host unit | |
KR102645983B1 (en) | Open channel vector command execution | |
WO2013092121A1 (en) | Solid-state storage management | |
CN111475438B (en) | IO request processing method and device for providing quality of service | |
US10740000B2 (en) | Adaptive transaction layer packet for latency balancing | |
CN109815170B (en) | Data redundancy processing method and related computer system | |
US11409466B2 (en) | Access control in CMB/PMR virtualization environment | |
US11199992B2 (en) | Automatic host buffer pointer pattern detection | |
CN108153582B (en) | IO command processing method and medium interface controller | |
CN108877862B (en) | Data organization of page stripes and method and device for writing data into page stripes | |
KR20230035811A (en) | Apparatus and method for controlling a shared memory in a data processing system | |
US20220050629A1 (en) | Completion management | |
KR20230036682A (en) | Memory device and memory system supporting interleaving operation and operation method thereof | |
CN118051181A (en) | Quality of service management in a memory subsystem |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: WESTERN DIGITAL TECHNOLOGIES, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:BENISTY, SHAY;REEL/FRAME:049647/0837 Effective date: 20190616 |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
AS | Assignment |
Owner name: JPMORGAN CHASE BANK, N.A., AS AGENT, ILLINOIS Free format text: SECURITY INTEREST;ASSIGNOR:WESTERN DIGITAL TECHNOLOGIES, INC.;REEL/FRAME:052915/0566 Effective date: 20200113 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
AS | Assignment |
Owner name: WESTERN DIGITAL TECHNOLOGIES, INC., CALIFORNIA Free format text: RELEASE OF SECURITY INTEREST AT REEL 052915 FRAME 0566;ASSIGNOR:JPMORGAN CHASE BANK, N.A.;REEL/FRAME:059127/0001 Effective date: 20220203 |
|
AS | Assignment |
Owner name: JPMORGAN CHASE BANK, N.A., ILLINOIS Free format text: PATENT COLLATERAL AGREEMENT - A&R LOAN AGREEMENT;ASSIGNOR:WESTERN DIGITAL TECHNOLOGIES, INC.;REEL/FRAME:064715/0001 Effective date: 20230818 Owner name: JPMORGAN CHASE BANK, N.A., ILLINOIS Free format text: PATENT COLLATERAL AGREEMENT - DDTL LOAN AGREEMENT;ASSIGNOR:WESTERN DIGITAL TECHNOLOGIES, INC.;REEL/FRAME:067045/0156 Effective date: 20230818 |
|
AS | Assignment |
Owner name: SANDISK TECHNOLOGIES, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:WESTERN DIGITAL TECHNOLOGIES, INC.;REEL/FRAME:067567/0682 Effective date: 20240503 |
|
AS | Assignment |
Owner name: SANDISK TECHNOLOGIES, INC., CALIFORNIA Free format text: CHANGE OF NAME;ASSIGNOR:SANDISK TECHNOLOGIES, INC.;REEL/FRAME:067982/0032 Effective date: 20240621 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |