CN116302105B - Access instruction scheduling method, system, hard disk, controller, storage medium and program product - Google Patents
Access instruction scheduling method, system, hard disk, controller, storage medium and program product Download PDFInfo
- Publication number
- CN116302105B CN116302105B CN202310266373.2A CN202310266373A CN116302105B CN 116302105 B CN116302105 B CN 116302105B CN 202310266373 A CN202310266373 A CN 202310266373A CN 116302105 B CN116302105 B CN 116302105B
- Authority
- CN
- China
- Prior art keywords
- instruction
- access
- flash memory
- priority information
- instruction scheduling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 52
- 230000015654 memory Effects 0.000 claims abstract description 189
- 239000007787 solid Substances 0.000 claims description 50
- 238000004891 communication Methods 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 5
- 230000002085 persistent effect Effects 0.000 description 17
- 230000008569 process Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 5
- 230000003068 static effect Effects 0.000 description 5
- 230000002093 peripheral effect Effects 0.000 description 4
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 238000013475 authorization Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011022 operating instruction Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/30—Arrangements for executing machine instructions, e.g. instruction decode
- G06F9/34—Addressing or accessing the instruction operand or the result ; Formation of operand address; Addressing modes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/30—Arrangements for executing machine instructions, e.g. instruction decode
- G06F9/38—Concurrent instruction execution, e.g. pipeline, look ahead
- G06F9/3836—Instruction issuing, e.g. dynamic instruction scheduling or out of order instruction execution
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The embodiment of the invention provides an access instruction scheduling method, an access instruction scheduling system, a hard disk, a controller, a storage medium and a program product. The access instruction scheduling method comprises the following steps: acquiring a flash memory access instruction of a host; analyzing the flash memory access instruction to obtain instruction scheduling priority information, wherein the instruction scheduling priority information is preconfigured based on the access delay requirement of the flash memory access instruction, and the higher the priority indicated by the instruction scheduling priority information is, the higher the access delay requirement of the flash memory access instruction is; and adding the flash memory access instruction into an instruction scheduling queue based on the instruction scheduling priority information. The scheme of the embodiment of the invention ensures the scheduling efficiency of the access instruction, improves the IO access efficiency of the data, and particularly greatly improves the IO access efficiency and the access load capacity in a multi-user high concurrency scene.
Description
Technical Field
The embodiment of the invention relates to the technical field of computers, in particular to an access instruction scheduling method, an access instruction scheduling system, a hard disk, a controller, a storage medium and a program product.
Background
Solid State Disk (SSD) is widely used for storing data in a server due to its superior read-write performance, and particularly in a cloud service scenario such as cloud computing or cloud storage, SSD is used for storing different types of data for different users.
However, in a concurrent access scenario such as a large number of users, there are a large number of IO access instructions to frequently write and/or read to a relatively limited solid state disk, and the conventional access instruction scheduling scheme cannot effectively manage the access latency requirement, which results in lower scheduling efficiency of the IO access instructions and lower efficiency of read and write operations.
Disclosure of Invention
In view of the above, embodiments of the present invention provide an access instruction scheduling method, system, hard disk, controller, storage medium and program product to at least partially solve the above-mentioned problems.
According to a first aspect of an embodiment of the present invention, there is provided an access instruction scheduling method, including: acquiring a flash memory access instruction of a host; analyzing the flash memory access instruction to obtain instruction scheduling priority information, wherein the instruction scheduling priority information is preconfigured based on the access delay requirement of the flash memory access instruction, and the higher the priority indicated by the instruction scheduling priority information is, the higher the access delay requirement of the flash memory access instruction is; and adding the flash memory access instruction into an instruction scheduling queue based on the instruction scheduling priority information.
In another implementation manner of the present invention, the parsing the flash access instruction to obtain instruction scheduling priority information includes: and analyzing a preset field in a protocol instruction format of the flash memory access instruction to obtain instruction scheduling priority information.
In another implementation manner of the present invention, the obtaining a flash memory access instruction of a host includes: and receiving a flash access instruction sent by a host through a flash access bus, wherein the protocol instruction format of the flash access instruction accords with the protocol of the flash access bus.
In another implementation of the present invention, the protocol of the flash access bus is NVMe protocol.
In another implementation manner of the present invention, the adding the flash access instruction to the instruction scheduling queue based on the instruction scheduling priority information includes: and adding the flash memory access instruction into an instruction scheduling queue indicated by the instruction scheduling priority information.
In another implementation manner of the present invention, the adding the flash access instruction to the instruction scheduling queue based on the instruction scheduling priority information includes: and adding the flash memory access instruction into a queue position indicated by the instruction scheduling priority information in an instruction scheduling queue.
In another implementation of the present invention, the method further includes: and when the current dispatching priority indicated by the instruction dispatching priority information is the current dispatching priority indicated by the instruction dispatching priority information, dispatching the instruction dispatching queue indicated by the instruction dispatching priority information.
In another implementation manner of the present invention, the scheduling the instruction scheduling queue indicated by the instruction scheduling priority information includes: and determining the access type information and the access logic address of each access instruction aiming at the flash memory in the instruction scheduling queue indicated by the instruction scheduling priority information.
According to a second aspect of an embodiment of the present invention, there is provided a solid state disk controller including: the acquisition module acquires a flash memory access instruction of the host; the analysis module analyzes the flash memory access instruction to obtain instruction scheduling priority information, wherein the instruction scheduling priority information is preconfigured based on the access delay requirement of the flash memory access instruction, and the higher the priority indicated by the instruction scheduling priority information is, the higher the access delay requirement of the flash memory access instruction is; and the queue module is used for adding the flash memory access instruction into an instruction scheduling queue based on the instruction scheduling priority information.
According to a third aspect of an embodiment of the present invention, there is provided a solid state disk, including: a flash memory; the solid state disk controller according to the second aspect is configured to access the flash memory based on a flash memory access instruction.
According to a fourth aspect of an embodiment of the present invention, there is provided an access instruction scheduling system including: the host generates instruction scheduling priority information based on the access delay requirement and configures the instruction scheduling priority information into flash memory access instructions of the flash memory; according to the solid state disk of the third aspect, the solid state disk obtains the flash memory access instruction from the host.
According to a fifth aspect of an embodiment of the present invention, there is provided a solid state disk controller including: the device comprises a processor, a memory, a communication interface and an internal bus, wherein the processor, the memory and the communication interface complete mutual communication through the internal bus; the memory is configured to store at least one executable instruction that causes the processor to perform operations corresponding to the method according to the first aspect.
According to a sixth aspect of embodiments of the present invention, there is provided a computer storage medium having stored thereon a computer program which, when executed by a processor, implements a method according to the first aspect.
According to a seventh aspect of embodiments of the present invention, there is provided a computer program product comprising computer instructions for instructing a computing device to perform the operations corresponding to the method according to the first aspect.
In the scheme of the embodiment of the invention, the pre-configured instruction scheduling priority information is obtained by analyzing the flash memory access instruction from the host, the instruction scheduling priority information is configured in the flash memory access instruction to be compatible with the traditional IO read-write process, the flash memory access instruction is added into the instruction scheduling queue based on the instruction scheduling priority information, no separate additional instruction scheduling information is needed, the scheduling efficiency of the access instruction is ensured, in addition, the higher the priority indicated by the instruction scheduling priority information is based on the access delay requirement pre-configured of the flash memory access instruction, the higher the access delay requirement of the flash memory access instruction is, therefore, the reliable delay management of the data with different access delay requirements is realized through the priority of the instruction scheduling, the IO access efficiency of the data is improved, and the IO access efficiency and the access load capacity are greatly improved especially in a multi-user high concurrency scene.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments described in the embodiments of the present invention, and other drawings may be obtained according to these drawings for a person having ordinary skill in the art.
Fig. 1A is a schematic diagram of an IO access architecture of a solid state disk SSD according to one example.
Fig. 1B is a software and hardware configuration block diagram of the solid state disk SSD illustrated in fig. 1A.
Fig. 2 is a flow chart of steps of a method for scheduling access instructions according to an embodiment of the present invention.
Fig. 3A is a schematic diagram of a software and hardware configuration of the embodiment of fig. 2.
Fig. 3B is an example of an instruction format of the IO access instruction of the embodiment of fig. 2.
Fig. 4 is a schematic structural diagram of an SSD controller according to another embodiment of the invention.
Detailed Description
In order to better understand the technical solutions in the embodiments of the present invention, the following description will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which are derived by a person skilled in the art based on the embodiments of the present invention, shall fall within the scope of protection of the embodiments of the present invention.
The implementation of the embodiments of the present invention will be further described below with reference to the accompanying drawings.
Fig. 1A is a schematic diagram of an IO access architecture of a solid state disk SSD according to one example. The IO access architecture of the present example includes a host 20 and a solid state disk SSD100, the solid state disk SSD100 including an SSD controller 104 and a memory 108.
The SSD controller 104 includes a side of the host 20, which is an IO access instruction side for accessing the solid state disk, and a side of the memory (flash NAND) 108, which corresponds to a logical address for reading and writing data. The device side refers to one side of the memory 108 for IO read-write data of the solid state disk, and corresponds to a physical address of the read-write data.
The SSD controller 104 further includes a flash memory middle layer (Flash translation layer, FTL) 130 as a software middle layer, so as to prevent protection of the flash memory, achieve balanced wear, increase the service time of the flash memory, and complete mapping from the logical address space of the host side to the physical address of the device side. The advantages and disadvantages of the FTL algorithm determine the performance, reliability, durability and the like of the solid state disk. With the development of SSD controllers, there are FTLs with combination of software and hardware to improve the management efficiency of flash memories.
In one example, FTL130 is implemented on the device side, employing the storage resources of the controller of the solid state disk. In another example, the FTL is implemented on the host side, including a processor and memory resource implementation of a computer in which the solid state disk resides. In the IO access architecture including the host 20, the SSD controller 104, and the flash memory (NAND flash) 108, the host 20 generates a read instruction or a write instruction, and transmits the IO access instruction, i.e., the read instruction or the write instruction, to the SSD controller 104.
SSD controller 104 may be configured with host interface circuitry 102, flash intermediate layer (FTL) 130, memory control circuitry 122, cache 14, and memory 15 such as Double Data Rate (DDR) memory. The host interface circuit 102 may be implemented as a Front End (FE) functional module for parsing and data transfer for the NVMe protocol. The memory control circuit 122 may BE implemented as a Back End (BE) for status management for the flash memory, and instruction parsing issues. The cache 14 may be a cache of software or hardware.
The software and hardware configuration of the solid state disk SSD in fig. 1A is further explained and illustrated with reference to fig. 1B. SSD controller 104 includes host interface circuitry 102, persistent memory controller circuitry 110, non-volatile block addressable memory controller circuitry 112, and, as examples of FTLs, processor 122, hardware assist circuitry 116, and static random access memory 130, processor 122 may cooperate with, for example, hardware assist circuitry 116, and static random access memory 130 to implement the functionality of the FTLs. As another example, SSD controller 104 may also be included in a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC).
The host system may communicate with the solid state disk 100 via a high speed serial computer expansion bus 120, such as a peripheral component interconnect express (PCIe) bus. The host interface circuit 102 manages communication via a peripheral component interconnect express (PCIe) bus. In some embodiments, the host system communicates via a peripheral component interconnect express (PCIe) bus using non-volatile memory express (NVMe) standard protocols.
The non-volatile memory express (NVMe) standard protocol defines a register level interface of host software to communicate with a Solid State Disk (SSD) 100 via a peripheral component interconnect express (PCIe) bus. The NVMe interface allows host software to communicate with the solid state disk 100 via the high speed serial computer expansion bus 120 based on a pairing mechanism of a commit queue and a completion queue. A commit queue (SQ) 144 and a Completion Queue (CQ) 142 are allocated in memory. Commit queue 144 is a circular buffer of fixed slot size used by host software to commit instructions for execution by SSD controller 104. Completion queue 142 is a circular buffer of fixed slot size for issuing the status of completed instructions.
The NVMe instruction set is used for paired commit queues and completion queues. The host software inserts instructions from the NVMe instruction set into the commit queue 144 for execution by the SSD controller 104. SSD controller 104 inserts the completion into associated completion queue 142.
Further, the solid state disk 100 further includes a persistent memory 106 and a nonvolatile memory 108 for use in the IO access process.
A portion of Static Random Access Memory (SRAM) 130 is allocated as L2P table cache 132 to store a portion of Persistent Memory (PM) L2P table 118 stored in persistent memory 106. Another portion of the sram 130 is allocated as a VPC cache 134 to store data transferred via the high speed serial computer expansion bus 120. In some embodiments, the static random access memory 130 is one million or Millions of Bytes (MBs).
Another portion of the sram 130 is also allocated as a physical-to-logical (P2L) table 136. Both the L2P table cache 132 and the physical-to-logical (P2L) table 136 store physical block addresses corresponding to logical block addresses in block-addressable non-volatile memory in the solid state disk. It should be appreciated that the entries in the P2L table 136 are static (not updated after they are written). Entries in the L2P table cache 132 are dynamically updated.
The L2P address table (also referred to as an L2P table) is stored in byte-addressable volatile memory, such as Dynamic Random Access Memory (DRAM) or Synchronous Dynamic Random Access Memory (SDRAM). The L2P table is stored in byte-addressable volatile memory to speed up reading physical block addresses from the L2P table to access physical blocks in block-addressable non-volatile memory (e.g., NAND flash memory) in the solid state disk. The byte-addressable volatile memory used to store the L2P table may be included in the solid state disk or in a host communicatively coupled to the solid state disk.
The L2P table may be stored in a host memory buffer, e.g., a portion of system DRAM. Each time an L2P table in the host memory buffer is written (updated), the L2P table stored in the block addressable non-volatile memory in the solid state disk is written simultaneously. The performance of writing to the L2P table is based on the longest write time, i.e., the time to write to the block-addressable nonvolatile memory or host memory buffer.
The L2P table may be stored in byte-addressable volatile memory in the solid state disk. However, the size of the L2P table depends on the user capacity of the solid state disk (e.g., about one Megabyte (MB) of user capacity per Gigabyte (GB)). The increase in non-volatile memory (also referred to as user capacity) in solid state drives requires a corresponding increase in byte-addressable volatile memory to store the L2P table.
The sram 130 is a volatile memory. Volatile memory is memory whose state (and thus the data stored therein) is uncertain when power to the device is interrupted. SRAM is a volatile memory type that uses latch circuits to store each bit. SRAM is typically used as a cache memory because data stored in SRAM does not need to be refreshed periodically, as compared to Dynamic Random Access Memory (DRAM).
The persistent memory 106 and the non-volatile memory 108 are non-volatile memories. A non-volatile memory (NVM) device is a memory whose state is determined even when the device power is interrupted. In one embodiment, the non-volatile memory 108 is a NAND flash memory, or more specifically a multi-threshold level NAND flash memory (e.g., single level cell ("SLC"), multi-level cell ("MLC"), three-level cell ("TLC"), four-level cell ("QLC"), five-level cell ("PLC"), or some other NAND flash memory).
The non-volatile memory 108 includes at least one non-volatile memory die (die), such as a NAND flash memory die. Typically, data is written (striped) across many NAND flash die in an SSD to optimize write bandwidth. The non-volatile memory on the non-volatile memory die includes a plurality of blocks, where each block includes a plurality of pages. Each of the plurality of pages is to store data and associated log content information. In some embodiments, the nonvolatile memory die has 2048 blocks, each block having 64 pages, and each page may store 2048 bytes of data and 64 bytes of log content information.
Persistent memory 106 is byte-addressable write-in-place nonvolatile memory. The hardware assist circuit 116 manages data transfer between the persistent memory 106 and the non-volatile memory 108. A portion of persistent memory 106 is allocated to store PM L2P table 118. A portion of the non-volatile memory 108 is used to store the main L2P table 148. The size of the PM L2P table 118 is based on a ratio of 1:1000 depending on the capacity of the non-volatile memory 108, e.g., the PM L2P table 118 in the persistent memory 106 is one Megabyte (MB) per Gigabyte (GB) of the non-volatile memory 108. PM L2P table 118 is a copy of main L2P table 148. Both the PM L2P table 118 and the primary L2P table 148 are stored in non-volatile memory and are periodically synchronized during operation of the solid state disk 100.
The remainder of the persistent memory 106 may be used to store the PM L2P table 118, user data, persistent memory log content information, and host log content information. In some embodiments, about 5% of the persistent memory is used to store the PM L2P table 118, about 5% is used to store persistent memory/host log content information, and about 90% is used to store user data.
In some embodiments, the user data stored in persistent memory 106 may include frequently accessed user data, operating system files, and executable programs. Frequently accessed user data may be referred to as "hot" user data. Data that is not accessed frequently (also referred to as "cold" user data) may be stored in the non-volatile memory 108. The persistent memory 106 may be referred to as a "storage accelerator memory" because the read latency of the persistent memory 106 is less than the read latency of the non-volatile memory 108.
In other embodiments, the persistent memory 106 may be SLC NAND and the block-addressable non-volatile memory 108 may be NAND with more than one bit per cell (e.g., MLC, TLC, QLC, PLC NAND). The read delay of SLC NAND is faster than that of NAND with more than one bit per cell.
Nonvolatile block-addressable memory controller circuitry 112 in SSD controller 104 queues instructions and processes the instructions (e.g., read, write ("program"), erase instructions for user data stored in nonvolatile memory 108).
In the instruction access architecture based on fig. 1A and fig. 1B, in a concurrent access scenario such as a large number of users, a large number of IO access instructions are present to frequently write and/or read to a relatively limited solid state disk, but a conventional access instruction scheduling scheme cannot be used to effectively manage access delay requirements, so that the scheduling efficiency of the IO access instructions is low and the efficiency of read-write operations is low.
Fig. 2 is a flow chart of steps of a method for scheduling access instructions according to an embodiment of the present invention. The access instruction scheduling method of the embodiment comprises the following steps:
S210: and acquiring a flash memory access instruction of the host.
It should be appreciated that the flash memory may be the flash memory 108 shown in fig. 1A and 1B, and that the flash access instructions may be sent from the host 20 to the host interface circuit 102, i.e., the flash access instructions (i.e., the IO read/write operation instructions) may be obtained by the host interface circuit 102. The flash access instruction may be a read operation instruction indicating a read access procedure and/or a write operation instruction indicating a write access procedure.
It should also be appreciated that the instruction dispatch priority information may be included in the flash access instruction by a host (e.g., one or more hosts), e.g., the host may respond to read and write operations in an application program, etc., in generating instruction dispatch priority information corresponding to the flash access instruction, that is, the instruction dispatch priority information reflects the read and write operations of the corresponding host.
S220: analyzing the flash memory access instruction to obtain instruction scheduling priority information, wherein the instruction scheduling priority information is preconfigured based on the access delay requirement of the flash memory access instruction, and the higher the priority indicated by the instruction scheduling priority information is, the higher the access delay requirement of the flash memory access instruction is.
It should be understood that, regarding the configuration manner of the instruction scheduling priority information, the instruction scheduling priority information is configured according to at least one influencing factor of the access delay requirement, where the at least one influencing factor includes, but is not limited to, the heat of the user data, the priority of the user to which the user data belongs, the service type of the user data, and the like. For example, in a multi-user concurrency scenario, the front-end browsing data of the user needs the background server to give feedback in time, otherwise, the user browsing process may be blocked, so that the front-end browsing data of the user belongs to data with higher heat, that is, the access delay requirement is higher. For another example, account data of the user and the like are used for privacy protection of the user, and the access delay requirement degree is low. In addition, the priority of the user to which the user data belongs may refer to a user level using an application program or the like, for example, the paid user level is higher than the general user level, and the priority of the user may refer to the management authority of the user, and the user level with a high management authority is equal to the user level with a low management authority, so that the higher the user level is, the higher the corresponding access delay requirement is, the lower the user level is, and the lower the corresponding access delay requirement is.
S230: and adding the flash memory access instruction into an instruction scheduling queue based on the instruction scheduling priority information.
It should be appreciated that the read and write instructions may be scheduled in unison based on the same instruction scheduling queue. The read operation queue and the write operation queue (i.e., queues formed by instructions of different access types) may also be scheduled based on different instruction scheduling queues. For example, the read operation queue and the write operation queue correspond to different priorities, so that different types of flash access instructions may be scheduled based on different instruction scheduling queues. The flash access instructions in the instruction dispatch queue may be from one or more hosts, for example, flash access instructions generated in response to a read operation and/or a write operation by each host.
In the scheme of the embodiment of the invention, the pre-configured instruction scheduling priority information is obtained by analyzing the flash memory access instruction from the host, the instruction scheduling priority information is configured in the flash memory access instruction to be compatible with the traditional IO read-write process, the flash memory access instruction is added into the instruction scheduling queue based on the instruction scheduling priority information, no separate additional instruction scheduling information is needed, the scheduling efficiency of the access instruction is ensured, in addition, the higher the priority indicated by the instruction scheduling priority information is based on the access delay requirement pre-configured of the flash memory access instruction, the higher the access delay requirement of the flash memory access instruction is, therefore, the reliable delay management of the data with different access delay requirements is realized through the priority of the instruction scheduling, the IO access efficiency of the data is improved, and the IO access efficiency and the access load capacity are greatly improved especially in a multi-user high concurrency scene.
In other examples, as an example of parsing the flash access instruction, a preset field in a protocol instruction format of the flash access instruction may be parsed to obtain the instruction scheduling priority information. For example, fig. 3B illustrates a protocol instruction format of a typical NVMe bus, a flash access instruction having 32 bits (bits), namely, bit 00:31. the fields of the instruction identifier correspond to bits 31:16, the fields of the memory locations (including physical pages (Physical Region Page, PRP) and linked lists (SGL)) correspond to bits 15:14, the reserved fields correspond to bits 13:10, the fields of the instruction fusion correspond to bits 09:08, and the instruction operation code corresponds to bits 07:00. Wherein the reserved field comprises 4 bits, which can represent 2 at most 4 =16 priorities. The pre-configured instruction scheduling priority information is configured to a preset field in a protocol instruction format, so that the existing flash memory access process and the instruction analysis process are greatly compatible.
Further, some bits of instruction scheduling priority information may refer to the popularity of user data as an influencing factor of access latency requirements, i.e. higher demand for access latency is required for higher popularity data. Other bits of the instruction scheduling priority information can indicate that a user corresponding to the user data is used as an influence factor of the access delay requirement, and the higher the priority of the user is, the higher the access delay requirement of the user data is. That is, the heat and user as a factor of influence of the different access latency requirements form a combination of the individual priorities of the instruction schedule.
In general, a combination of different factors affecting multiple access latency requirements forms the respective instruction dispatch priority. The priorities between the various influencing factors (which are different from the instruction scheduling priorities) may also be different, for example, in the above example, if the user data has a higher priority than the influencing factors, some bits indicating the user data have a higher priority reference weight than some bits indicating the user. For any two of the influencing factors of the access latency requirement, among the respective bits of the instruction scheduling priority information, at least one bit (indicating one influencing factor) arranged earlier in the parsing order of the instruction scheduling priority information has a higher priority reference weight than at least one bit (indicating another influencing factor) arranged later in the parsing order, for example, if parsed in a left-to-right manner in fig. 3B, bit 13 is bit 13:10: 12 is greater than the influencing factor priority of bits 11:10. In this case, if bit 13:12 indicates the user and bits 11:10 indicates the user data, indicating that the user has a greater impact weight on the instruction parsing order than the user data, in other words, if the instruction scheduling weight of the user is slightly greater than the instruction scheduling weight of the user data, this may be achieved by ordering the user in a position further forward than the parsing order of the user data. The instruction scheduling priority information configured in the mode correlates the instruction scheduling efficiency affecting the instruction analysis efficiency with the instruction scheduling priority, and further improves the instruction scheduling efficiency.
In other words, the host interface 310, such as the host interface circuit 102, receives the flash access instructions sent with the host over the flash access bus, and then parses the flash access instructions, which may have a protocol instruction format that conforms to the protocol of the flash access bus. For example, as shown in fig. 3A, by receiving a flash access instruction through a host interface 310, such as host interface circuit 102, host interface 310 may also parse the flash access instruction for the reserved field to obtain instruction dispatch priority information. The host interface 310 may then transmit the instruction dispatch priority information to the priority parser 320, where the priority parser 320 determines the dispatch priority indicated by the instruction dispatch priority information and adds the flash access instruction to the instruction dispatch queue.
Returning to the configuration process example of the instruction scheduling priority information, in the process of analyzing the flash memory access instruction, the priority analyzer 320 analyzes the bit in the front of the analysis order and then analyzes the bit in the back of the analysis order, so that the instruction scheduling efficiency affecting the instruction analysis efficiency is associated with the instruction scheduling priority, and the instruction scheduling efficiency is further improved.
Accordingly, the instruction scheduler 330 performs scheduling and distribution of flash access instructions when the indicated scheduling priority corresponds to the current scheduling priority in the priority sequence. Without loss of generality, the instruction dispatch queue indicated by the instruction dispatch priority information is dispatched at the current dispatch priority indicated by the instruction dispatch priority information. Specifically, as an example of scheduling the instruction scheduling queue indicated by the instruction scheduling priority information, the access type information and the access logic address of each access instruction for the flash memory in the instruction scheduling queue indicated by the instruction scheduling priority information may be determined, then, continuing in the FTL, if the access type indicated by the instruction identification is write access, the access physical address and the access logic address of the flash memory are associated into the L2P table, and if the access type is read access, the access physical address of the flash memory is determined directly based on the L2P table stored in advance, as described in fig. 1A and 1B.
For a specific implementation of adding the flash access instruction to the instruction dispatch queue based on the instruction dispatch priority information, the flash access instruction may be added to the instruction dispatch queue indicated by the instruction dispatch priority information. That is, in this case, the instruction scheduler 330 may manage a plurality of instruction scheduling queues, different instruction scheduling queues corresponding to different instruction scheduling priority information, for example, different instruction scheduling queues corresponding to different priorities or priority ranges, and as an example, a flash access instruction may be added to the end of queue of the instruction scheduling queue corresponding to the instruction scheduling priority information. It should be appreciated that in this case, each instruction dispatch queue may be a first-in-first-out queue.
In another example, based on the instruction dispatch priority information, adding the flash access instruction to the instruction dispatch queue includes: and adding the flash memory access instruction into a queue position indicated by the instruction scheduling priority information in the instruction scheduling queue. That is, the instruction scheduler 330 may manage a single instruction scheduling queue, the head of the instruction scheduling queue indicates a higher priority, the tail of the instruction scheduling queue indicates a lower priority, the middle position of the instruction scheduling queue indicates a middle priority, the instruction scheduling queue may be divided into a plurality of position ranges from the head of the queue to the tail of the queue, corresponding to a plurality of priorities from high to low, respectively, and then the flash access instruction may be added to the queue position in the instruction scheduling queue. In this case, the instruction dispatch queue may be formed as a linked list with the queue location corresponding to the virtual memory address of the linked list.
The solid state disk controller according to the embodiment of the present invention will be described with reference to fig. 4. The solid state disk controller includes:
the obtaining module 410 obtains a flash access instruction of the host.
The analyzing module 420 analyzes the flash memory access instruction to obtain instruction scheduling priority information, wherein the instruction scheduling priority information is preconfigured based on the access delay requirement of the flash memory access instruction, and the higher the priority indicated by the instruction scheduling priority information is, the higher the access delay requirement of the flash memory access instruction is.
The queue module 430 adds the flash access instruction to an instruction dispatch queue based on the instruction dispatch priority information.
In the scheme of the embodiment of the invention, the pre-configured instruction scheduling priority information is obtained by analyzing the flash memory access instruction from the host, the instruction scheduling priority information is configured in the flash memory access instruction to be compatible with the traditional IO read-write process, the flash memory access instruction is added into the instruction scheduling queue based on the instruction scheduling priority information, no separate additional instruction scheduling information is needed, the scheduling efficiency of the access instruction is ensured, in addition, the higher the priority indicated by the instruction scheduling priority information is based on the access delay requirement pre-configured of the flash memory access instruction, the higher the access delay requirement of the flash memory access instruction is, therefore, the reliable delay management of the data with different access delay requirements is realized through the priority of the instruction scheduling, the IO access efficiency of the data is improved, and the IO access efficiency and the access load capacity are greatly improved especially in a multi-user high concurrency scene.
In other examples, the parsing module is specifically configured to: and analyzing a preset field in a protocol instruction format of the flash memory access instruction to obtain instruction scheduling priority information.
In other examples, the acquisition module is specifically configured to: and receiving a flash access instruction sent by a host through a flash access bus, wherein the protocol instruction format of the flash access instruction accords with the protocol of the flash access bus.
In other examples, the protocol of the flash access bus is NVMe protocol.
In other examples, the queue module is specifically configured to: and adding the flash memory access instruction into an instruction scheduling queue indicated by the instruction scheduling priority information.
In other examples, the queue module is specifically configured to: and adding the flash memory access instruction into a queue position indicated by the instruction scheduling priority information in an instruction scheduling queue.
In other examples, the solid state disk controller further includes: and the scheduling module schedules the instruction scheduling queue indicated by the instruction scheduling priority information when the current scheduling priority indicated by the instruction scheduling priority information.
In other examples, the scheduling module is specifically configured to: and determining the access type information and the access logic address of each access instruction aiming at the flash memory in the instruction scheduling queue indicated by the instruction scheduling priority information.
Another embodiment of the present invention further provides a solid state disk, which may be a solid state disk 100 as shown in fig. 1A and fig. 1B. The solid state disk 100 includes a flash memory and a solid state disk controller in the embodiment of fig. 4, where the solid state disk controller is configured to access the flash memory based on a flash memory access instruction.
The invention also provides an access instruction scheduling system. The access instruction scheduling system of the present embodiment may include a host 20 and a solid state disk 100 as shown in fig. 1A and 1B. The host 20 generates instruction scheduling priority information based on the access latency requirements and configures the instruction scheduling priority information into flash access instructions of the flash memory. The solid state disk 100 obtains the flash access instruction from the host 20.
An SSD controller of another embodiment of the invention may include: a processor (processor) 122 for executing programs, a communication interface (Communications Interface) 112, a memory (memory) 130, and an internal bus, for example, as described in fig. 1A and 1B.
The processor 122, communication interface 112, and memory communicate with each other via an internal bus.
A communication interface 112 for communicating with the memory 108.
The processor 122 is configured to execute a program, and may specifically perform the relevant steps in the foregoing embodiments.
In particular, the program may include program code including computer-operating instructions.
Further, the processor 122 may be a CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors comprised by the smart device may be the same type of processor, such as one or more CPUs; but may also be different types of processors such as one or more CPUs and one or more ASICs.
In addition, the memory 130 may also be used to store programs. The memory may comprise high-speed RAM memory or may further comprise non-volatile memory, such as at least one disk memory.
The program may include a plurality of computer instructions, and the program may specifically enable the processor to execute operations corresponding to the access instruction scheduling method of fig. 2 described in any one of the foregoing method embodiments through the plurality of computer instructions.
The specific implementation of each step in the above procedure may refer to the corresponding step and corresponding description in the unit in the above method embodiment, and have corresponding beneficial effects, which are not repeated herein. It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the apparatus and modules described above may refer to corresponding procedure descriptions in the foregoing method embodiments, which are not repeated herein.
The embodiment of the invention also provides a computer storage medium, on which a computer program is stored, which when executed by a processor, implements the access instruction scheduling method described in any one of the foregoing method embodiments. The computer storage media includes, but is not limited to: a compact disk read Only (Compact Disc Read-Only Memory, CD-ROM), random access Memory (Random Access Memory, RAM), floppy disk, hard disk, magneto-optical disk, or the like.
The embodiment of the invention also provides a computer program product, which comprises computer instructions, wherein the computer instructions instruct a computing device to execute the operations corresponding to any one of the access instruction scheduling methods in the method embodiments.
In addition, it should be noted that, the information related to the user (including, but not limited to, user equipment information, user personal information, etc.) and the data related to the embodiment of the present invention (including, but not limited to, sample data for training the model, data for analyzing, stored data, presented data, etc.) are information and data authorized by the user or sufficiently authorized by each party, and the collection, use and processing of the related data need to comply with the related laws and regulations and standards of the related country and region, and provide a corresponding operation entry for the user to select authorization or rejection.
It should be noted that, according to implementation requirements, each component/step described in the embodiments of the present invention may be split into more components/steps, or two or more components/steps or part of operations of the components/steps may be combined into new components/steps, so as to achieve the objects of the embodiments of the present invention.
The methods according to embodiments of the present invention described above may be implemented in hardware, firmware, or as software or computer code storable in a recording medium such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk, or as computer code originally stored in a remote recording medium or a non-transitory machine-readable medium and to be stored in a local recording medium downloaded through a network, so that the methods described herein may be processed by such software on a recording medium using a general purpose computer, a special purpose processor, or programmable or special purpose hardware such as an application specific integrated circuit (Application Specific Integrated Circuit, ASIC) or field programmable or gate array (Field Programmable Gate Array, FPGA). It is understood that a computer, processor, microprocessor controller, or programmable hardware includes a Memory component (e.g., random access Memory (Random Access Memory, RAM), read-Only Memory (ROM), flash Memory, etc.) that can store or receive software or computer code that, when accessed and executed by the computer, processor, or hardware, performs the methods described herein. Furthermore, when a general purpose computer accesses code for implementing the methods illustrated herein, execution of the code converts the general purpose computer into a special purpose computer for performing the methods illustrated herein.
Those of ordinary skill in the art will appreciate that the elements and method steps of the examples described in connection with the embodiments disclosed herein can be implemented as electronic hardware, or as a combination of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the embodiments of the present invention.
The above embodiments are only for illustrating the embodiments of the present invention, but not for limiting the embodiments of the present invention, and various changes and modifications may be made by one skilled in the relevant art without departing from the spirit and scope of the embodiments of the present invention, so that all equivalent technical solutions also fall within the scope of the embodiments of the present invention, and the scope of the embodiments of the present invention should be defined by the claims.
Claims (11)
1. An access instruction scheduling method, comprising:
acquiring a flash memory access instruction of a host, wherein a protocol instruction format of the flash memory access instruction accords with an NVMe protocol;
analyzing a preset field in a protocol instruction format of the flash memory access instruction to obtain instruction scheduling priority information, wherein in each bit of the instruction scheduling priority information, at least one bit arranged in front of an analysis order of the instruction scheduling priority information has higher priority reference weight than at least one bit arranged in back of the analysis order, the instruction scheduling priority information is preconfigured based on the access delay requirement of the flash memory access instruction, wherein the higher the priority indicated by the instruction scheduling priority information is, the higher the access delay requirement of the flash memory access instruction is, and for any two of a plurality of influencing factors of the access delay requirements, the at least one bit arranged in front of the analysis order is used for indicating one of the two access delay requirements, and the at least one bit arranged in front of the analysis order is used for indicating the other of the two access delay requirements, and the plurality of influencing factors comprise the heat degree of user data, the priority of a user to which the user data belongs and the service type of the user data;
And adding the flash memory access instruction into an instruction scheduling queue based on the instruction scheduling priority information.
2. The method of claim 1, wherein the obtaining flash access instructions of the host comprises:
and receiving a flash memory access instruction sent by the host through a flash memory access bus.
3. The method of claim 1, wherein the adding the flash access instruction to an instruction dispatch queue based on the instruction dispatch priority information comprises:
and adding the flash memory access instruction into an instruction scheduling queue indicated by the instruction scheduling priority information.
4. The method of claim 1, wherein the adding the flash access instruction to an instruction dispatch queue based on the instruction dispatch priority information comprises:
and adding the flash memory access instruction into a queue position indicated by the instruction scheduling priority information in an instruction scheduling queue.
5. The method of claim 1, wherein the method further comprises:
and when the current dispatching priority indicated by the instruction dispatching priority information is the current dispatching priority indicated by the instruction dispatching priority information, dispatching the instruction dispatching queue indicated by the instruction dispatching priority information.
6. The method of claim 5, wherein the scheduling the instruction dispatch queue indicated by the instruction dispatch priority information comprises:
and determining the access type information and the access logic address of each access instruction aiming at the flash memory in the instruction scheduling queue indicated by the instruction scheduling priority information.
7. A solid state disk controller comprising:
the acquisition module acquires a flash memory access instruction of the host, wherein a protocol instruction format of the flash memory access instruction accords with an NVMe protocol;
the analyzing module analyzes a preset field in a protocol instruction format of the flash memory access instruction to obtain instruction scheduling priority information, wherein in each bit of the instruction scheduling priority information, at least one bit arranged in front of an analysis order of the instruction scheduling priority information has higher priority reference weight than at least one bit arranged in back of the analysis order, the instruction scheduling priority information is preconfigured based on the access delay requirement of the flash memory access instruction, wherein the higher the priority indicated by the instruction scheduling priority information is, the higher the access delay requirement of the flash memory access instruction is, and for any two of multiple influence factors of the access delay requirement, the at least one bit arranged in front of the analysis order indicates one of the access delay requirements, and the at least one bit arranged in front of the analysis order indicates the other of the access delay requirements, and the multiple influence factors comprise the degree of user data, the priority of a user to which the user data belongs, and the service type of the user data;
And the queue module is used for adding the flash memory access instruction into an instruction scheduling queue based on the instruction scheduling priority information.
8. A solid state disk comprising:
a flash memory;
the solid state disk controller of claim 7, the solid state disk controller to access the flash memory based on a flash memory access instruction.
9. An access instruction scheduling system comprising:
the host generates instruction scheduling priority information based on the access delay requirement and configures the instruction scheduling priority information into flash memory access instructions of the flash memory;
the solid state disk of claim 8, the solid state disk obtaining the flash access instruction from the host.
10. A solid state disk controller comprising: the device comprises a processor, a memory, a communication interface and an internal bus, wherein the processor, the memory and the communication interface complete mutual communication through the internal bus;
the memory is configured to store at least one executable instruction that causes the processor to perform operations corresponding to the method according to any one of claims 1-6.
11. A computer storage medium having stored thereon a computer program which, when executed by a processor, implements the method according to any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310266373.2A CN116302105B (en) | 2023-03-17 | 2023-03-17 | Access instruction scheduling method, system, hard disk, controller, storage medium and program product |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310266373.2A CN116302105B (en) | 2023-03-17 | 2023-03-17 | Access instruction scheduling method, system, hard disk, controller, storage medium and program product |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116302105A CN116302105A (en) | 2023-06-23 |
| CN116302105B true CN116302105B (en) | 2024-03-12 |
Family
ID=86820363
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310266373.2A Active CN116302105B (en) | 2023-03-17 | 2023-03-17 | Access instruction scheduling method, system, hard disk, controller, storage medium and program product |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116302105B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117270726B (en) * | 2023-11-20 | 2024-04-19 | 荣耀终端有限公司 | Information interaction method, host and terminal equipment |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8341300B1 (en) * | 2007-08-30 | 2012-12-25 | Virident Systems, Inc. | Systems for sustained read and write performance with non-volatile memory |
| CN107133100A (en) * | 2017-04-26 | 2017-09-05 | 新华三技术有限公司 | Storage system service quality Q oS control methods and device |
| CN107967224A (en) * | 2017-10-12 | 2018-04-27 | 记忆科技(深圳)有限公司 | A kind of method of solid state hard disc improving performance uniformity |
| CN112395239A (en) * | 2019-08-14 | 2021-02-23 | 英特尔公司 | Techniques for latency-based service level agreement management in remote direct memory access networks |
| CN113821175A (en) * | 2021-09-27 | 2021-12-21 | 山东华芯半导体有限公司 | SSD instruction scheduling method and system based on storage content priority |
| CN114138683A (en) * | 2021-12-02 | 2022-03-04 | 湖南国科微电子股份有限公司 | Host memory access method and device, electronic equipment and storage medium |
| CN114153578A (en) * | 2021-11-19 | 2022-03-08 | 中汽创智科技有限公司 | Data access method, device, equipment and storage medium |
| CN115357377A (en) * | 2022-07-25 | 2022-11-18 | 芯来智融半导体科技(上海)有限公司 | Memory control scheduling method and device, computer equipment and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10452278B2 (en) * | 2017-03-24 | 2019-10-22 | Western Digital Technologies, Inc. | System and method for adaptive early completion posting using controller memory buffer |
-
2023
- 2023-03-17 CN CN202310266373.2A patent/CN116302105B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8341300B1 (en) * | 2007-08-30 | 2012-12-25 | Virident Systems, Inc. | Systems for sustained read and write performance with non-volatile memory |
| CN107133100A (en) * | 2017-04-26 | 2017-09-05 | 新华三技术有限公司 | Storage system service quality Q oS control methods and device |
| CN107967224A (en) * | 2017-10-12 | 2018-04-27 | 记忆科技(深圳)有限公司 | A kind of method of solid state hard disc improving performance uniformity |
| CN112395239A (en) * | 2019-08-14 | 2021-02-23 | 英特尔公司 | Techniques for latency-based service level agreement management in remote direct memory access networks |
| CN113821175A (en) * | 2021-09-27 | 2021-12-21 | 山东华芯半导体有限公司 | SSD instruction scheduling method and system based on storage content priority |
| CN114153578A (en) * | 2021-11-19 | 2022-03-08 | 中汽创智科技有限公司 | Data access method, device, equipment and storage medium |
| CN114138683A (en) * | 2021-12-02 | 2022-03-04 | 湖南国科微电子股份有限公司 | Host memory access method and device, electronic equipment and storage medium |
| CN115357377A (en) * | 2022-07-25 | 2022-11-18 | 芯来智融半导体科技(上海)有限公司 | Memory control scheduling method and device, computer equipment and storage medium |
Non-Patent Citations (4)
| Title |
|---|
| Efficient Buffer Management for Tree Indexes on Solid State Drives;Yoon, Joohyeong;《IEEE》;20190101;全文 * |
| NVMe存储协议浅论;郝嘉;候梦清;;信息系统工程(第03期);全文 * |
| 基于NVMe over TCP的分离式存储系统关键技术研究;乔星涵;《CNKI优秀硕士学位论文全文库（信息科技辑）》;20211001;全文 * |
| 基于高性能I/O技术的Memcached优化研究;安仲奇;杜昊;李强;霍志刚;马捷;;计算机研究与发展(第04期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116302105A (en) | 2023-06-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20080195833A1 (en) | Systems, methods and computer program products for operating a data processing system in which a file system's unit of memory allocation is coordinated with a storage system's read/write operation unit | |
| US20150253992A1 (en) | Memory system and control method | |
| CN107122130B (en) | Data deduplication method and device | |
| CN107203330B (en) | Read-write data stream oriented flash memory data distribution method | |
| CN116302105B (en) | Access instruction scheduling method, system, hard disk, controller, storage medium and program product | |
| US20220197563A1 (en) | Qos traffic class latency model for just-in-time (jit) schedulers | |
| US11520520B2 (en) | Memory system and method of controlling nonvolatile memory | |
| CN111796759A (en) | Computer readable storage medium and method for fragment data reading on multiple planes | |
| CN111563052A (en) | Cache method and device for reducing read delay, computer equipment and storage medium | |
| CN104252422A (en) | Memory access method and memory controller | |
| CN109799959A (en) | A method of it improving open channel solid-state disk and writes concurrency | |
| CN116361232A (en) | Processing method and device for on-chip cache, chip and storage medium | |
| WO2021035555A1 (en) | Data storage method and apparatus for solid state disk and solid state disk (ssd) | |
| CN111176566B (en) | eMMC read-write control method supporting queue command and storage medium | |
| EP3834072B1 (en) | Controller command scheduling in a memory system to increase command bus utilization | |
| CN113448511A (en) | Sequential prefetching through a linked array | |
| US11972110B2 (en) | Storage device and storage system | |
| US20200285413A1 (en) | Memory controller | |
| US20150212759A1 (en) | Storage device with multiple processing units and data processing method | |
| CN115938428A (en) | Controller for dynamic random access memory, control method and electronic equipment | |
| US11494113B2 (en) | Computer program product and method and apparatus for scheduling execution of host commands | |
| CN114625307A (en) | Computer readable storage medium, and data reading method and device of flash memory chip | |
| CN112099731A (en) | Data storage device and data processing method | |
| US11868287B2 (en) | Just-in-time (JIT) scheduler for memory subsystems | |
| TWI758745B (en) | Computer program product and method and apparatus for scheduling executions of host commands |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |