US11314438B2 - Arithmetic processing device, information processing device, and control method for arithmetic processing device - Google Patents
Arithmetic processing device, information processing device, and control method for arithmetic processing device Download PDFInfo
- Publication number
- US11314438B2 US11314438B2 US16/695,282 US201916695282A US11314438B2 US 11314438 B2 US11314438 B2 US 11314438B2 US 201916695282 A US201916695282 A US 201916695282A US 11314438 B2 US11314438 B2 US 11314438B2
- Authority
- US
- United States
- Prior art keywords
- read
- request
- address
- memory
- memory module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active, expires
Links
- 238000000034 method Methods 0.000 title claims description 5
- 230000010365 information processing Effects 0.000 title claims description 4
- 230000015654 memory Effects 0.000 claims abstract description 352
- 230000004044 response Effects 0.000 claims abstract description 61
- 230000005540 biological transmission Effects 0.000 claims abstract description 23
- 230000002123 temporal effect Effects 0.000 description 5
- 230000009467 reduction Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0877—Cache access modes
- G06F12/0882—Page mode
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0806—Multiuser, multiprocessor or multiprocessing cache systems
- G06F12/0815—Cache consistency protocols
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0866—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches for peripheral storage systems, e.g. disk cache
- G06F12/0868—Data transfer between cache memory and other subsystems, e.g. storage devices or host systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
- G06F3/0611—Improving I/O performance in relation to response time
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0679—Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/30—Arrangements for executing machine instructions, e.g. instruction decode
- G06F9/30003—Arrangements for executing specific machine instructions
- G06F9/30007—Arrangements for executing specific machine instructions to perform operations on data operands
- G06F9/3001—Arithmetic instructions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/30—Arrangements for executing machine instructions, e.g. instruction decode
- G06F9/38—Concurrent instruction execution, e.g. pipeline or look ahead
- G06F9/3836—Instruction issuing, e.g. dynamic instruction scheduling or out of order instruction execution
- G06F9/3842—Speculative instruction execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/0223—User address space allocation, e.g. contiguous or non contiguous base addressing
- G06F12/023—Free address space management
- G06F12/0238—Memory management in non-volatile memory, e.g. resistive RAM or ferroelectric memory
- G06F12/0246—Memory management in non-volatile memory, e.g. resistive RAM or ferroelectric memory in block erasable memory, e.g. flash memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0844—Multiple simultaneous or quasi-simultaneous cache accessing
- G06F12/0855—Overlapped cache accessing, e.g. pipeline
- G06F12/0859—Overlapped cache accessing, e.g. pipeline with reload from main memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
- G06F2212/10—Providing a specific technical effect
- G06F2212/1016—Performance improvement
- G06F2212/1024—Latency reduction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
- G06F2212/21—Employing a record carrier using a specific recording technology
- G06F2212/214—Solid state disk
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
- G06F2212/31—Providing disk cache in a specific location of a storage system
- G06F2212/313—In storage device
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
- G06F2212/50—Control mechanisms for virtual memory, cache or TLB
- G06F2212/507—Control mechanisms for virtual memory, cache or TLB using speculative control
Definitions
- the present invention relates to an arithmetic processing device, an information processing device, and a control method for the arithmetic processing device.
- a DRAM module which is a volatile memory, is mainly used as the main storage device of a personal computer or a server.
- a secondary storage device such as a solid state drive (SSD) or a hard disk drive (HDD), for example, is expanded in the main storage device to reduce access to the secondary storage, have come to attention.
- SSD solid state drive
- HDD hard disk drive
- a memory element having a larger storage capacity than a DRAM is mounted in a memory module of the DRAM.
- One of these devices is a device in which a volatile memory such as a DRAM, which has a high unit price but can be accessed at high speed, and a nonvolatile memory such as a NAND type flash memory, which has a low unit price and a large capacity, are mounted in a single memory module.
- a nonvolatile dual in line memory module NVDIMM.
- This memory module is a large-capacity, nonvolatile memory module in which a DRAM is used as a cache.
- a nonvolatile memory module mounting both a volatile memory and a nonvolatile memory, such as an NVDIMM
- the DRAM which serves as the volatile memory
- the cache of the large-capacity nonvolatile memory.
- the nonvolatile memory module transmits data to the host out of order, i.e. in a different order to the read order.
- handshakes are performed between the host and the memory module to exchange signals between reading and data acquisition, and therefore the latency is longer than that of the memory module of the DRAM.
- a speculative read in which data can be acquired without handshakes and therefore a latency is shorter than normal reads, can be used.
- the latency is shorter than that of a normal read when the DRAM cache is hit but longer than that of a normal read when the DRAM cache is missed.
- One aspect of an embodiment is an arithmetic processing device comprising: an arithmetic circuit configured to issue a read request to a memory; and a memory access controller configured to perform access control corresponding to the read request from the arithmetic circuit on a memory module having a volatile memory, which has a first capacity and is able to read with a first latency, and a nonvolatile memory, which has a second capacity that is larger than the first capacity and is able to read with a second latency that is longer than the first latency, the volatile memory operating as a cache of the nonvolatile memory, wherein the memory access controller is configured to:
- FIG. 1 is a view depicting example configurations of an arithmetic processing device and a nonvolatile memory module according to this embodiment.
- FIG. 2 is a view depicting an operation sequence executed in response to a normal read.
- FIG. 3 is a view depicting timing charts of signals exchanged between the memory access controller and the nonvolatile memory module in response to a normal read.
- FIG. 4 is a view depicting an operation sequence executed in response to a normal read.
- FIG. 5 is a view depicting timing charts of signals exchanged between the memory access controller and the nonvolatile memory module in response to a normal read.
- FIG. 6 is a view depicting an operation sequence executed in response to a speculative read.
- FIG. 7 is a view depicting timing charts of signals exchanged between the memory access controller and the nonvolatile memory module in response to a speculative read.
- FIG. 8 is a view depicting an operation sequence executed in response to a speculative read.
- FIG. 9 is a view depicting a timing chart of signals exchanged between the memory access controller and the nonvolatile memory module during in response to the speculative read.
- FIG. 10 is a view of a table illustrating examples of the latency of the nonvolatile memory module.
- FIG. 11 is a view depicting a relationship between the read data transmitted to the nonvolatile memory module and the two memories thereof.
- FIG. 12 is a view depicting an example configuration of the memory access controller MAC and the nonvolatile memory module NV_MM according to this embodiment.
- FIG. 13 is a view depicting an example configuration of the command issuance unit CMD_ISSU in a memory access controller according to a first embodiment.
- FIG. 14 is a view depicting a flowchart of an example operation of the command issuance unit CMD_ISSU.
- FIG. 15 is a view depicting a specific example of the operations performed by the command issuance unit in cases 1 and 2 .
- FIG. 16 is a view depicting an example configuration of the command issuance unit CMD_ISSU provided in the memory access controller according to the second embodiment.
- FIG. 17 is a view depicting a flowchart of an example operation of the command issuance unit CMD_ISSU.
- FIG. 18 is a view depicting specific examples of operations performed by the command issuance unit in a plurality of cases.
- FIG. 19 is a view depicting an example operation in a case where the memory access controller updates the page address table every time a miss occurs.
- FIG. 20 is a view depicting an example operation in which the memory access controller updates the page address table once every N read requests.
- FIG. 21 is a view depicting a comparison between a case in which either only the normal read XREAD is issued or only the speculative read SREAD is issued and a case such as that of the second embodiment, in which either XREAD or SREAD is issued depending on whether a hit is obtained on the page address table.
- FIG. 1 is a view depicting example configurations of an arithmetic processing device and a nonvolatile memory module according to this embodiment.
- An information processing device 1 includes an arithmetic processing device (a processor or a Central Processor Unit (CPU); referred to hereafter as the processor) CPU and a nonvolatile memory module NV_MM.
- the processor CPU uses the nonvolatile memory module as a main memory.
- the processor CPU includes arithmetic cores CORE_ 0 to CORE_n ⁇ 1 serving as n (n being a plurality) arithmetic circuits, each having an operator 2 and a level 1 cache L1_CACHE, a level 2 cache L2_CACHE provided in common for the plurality of arithmetic cores (referred to simply as “the cores” hereafter), and a memory access controller MAC.
- the memory access controller may be constituted by a different chip to the processor CPU. Moreover, a single or a plurality of the processor CPU may be provided.
- the nonvolatile memory module MV_MM includes a memory controller M_CON, a volatile memory V_MEM such as a DRAM, and a nonvolatile memory NV_MEM such as a NAND type flash memory.
- the volatile memory V_MEM is used as a cache of the nonvolatile memory NV_MEM. Accordingly, a cache tag is stored in the volatile memory V_MEM.
- an instruction issuing unit not depicted in the figure, including an instruction fetcher and an instruction decoder issues instructions, and the operator 2 executes the issued instructions.
- the core issues a memory request requesting access to the memory.
- Memory requests include read requests and write requests.
- the L1 cache L1_CACHE executes the memory request, which includes reading data from or writing data to the L1 cache, and when the memory request is a cache miss, a further memory request is issued to the L2 cache L2_CACHE.
- the L2 cache L2_CACHE executes the memory requests with respect to the data in the L2 cache when the issued memory request is a cache hit and issues a further memory request to the memory access controller MAC when the memory request is a cache miss.
- the nonvolatile memory module NV_MM includes the memory controller M_CON, the volatile memory V_MEM, such as a DRAM, and the nonvolatile memory NV_MEM, such as a NAND type flash memory.
- the volatile memory V_MEM is used as the cache of the nonvolatile memory NV_MEM.
- the read latency of the volatile memory V_MEM is shorter than that of the nonvolatile memory NV_MEM, whereas the data capacity of the volatile memory is smaller than that of the nonvolatile memory.
- reading to the nonvolatile memory module is performed in byte units (one byte or a plurality of bytes). Reading to the volatile memory V_MEM is also performed in byte units. Reading to the nonvolatile memory NV_MEM, however, is performed in page units, which have a larger capacity than byte units.
- the memory controller M_CON transmits the byte unit data at the read destination in the volatile memory V_MEM to the memory access controller MAC.
- a cache miss is obtained, on the other hand, data in a page area including the read address within the nonvolatile memory NV_MEM are registered (stored) in the volatile memory V_MEM, whereupon the data in the page area corresponding to the read address are transmitted to the memory access controller MAC.
- the volatile memory V_MEM is a DRAM
- the read latency similarly to that of a DRAM, is short. Therefore, when a cache hit is obtained, the nonvolatile memory module transmits the data in the volatile memory V_MEM to the memory access controller, and as a result, a similar reduction in read latency to that achieved with a DRAM is realized.
- the nonvolatile memory module transmits the data in the nonvolatile memory NV_MEM to the memory access controller MAC, and as a result, the read latency becomes longer than that of a DRAM.
- the nonvolatile memory module NV_MM is configured to support two types of reads, namely normal reads and speculative reads.
- a normal read is a non-speculative read, i.e. not a speculative read.
- the volatile memory V_MEM is used as the cache of the nonvolatile memory NV_MEM, and therefore the nonvolatile memory module NV_MM includes storage means equivalent to a cache tag for registering address information relating to the data stored in the volatile memory, for example.
- the cache tag is provided in the volatile memory V_MEM, for example, but may be provided in the memory controller M_CON.
- address information (information indicating a predetermined unit address or information indicating a page address) of the page area including the read destination address is registered in the cache tag, for example. Note that requirements relating to the cache control described above are not included in the intended scope of the volatile memory module.
- FIGS. 2 and 4 are views depicting an operation sequence executed in response to a normal read.
- a normal read command is denoted as XREAD, for example.
- FIGS. 2 and 4 depict the memory access controller MAC, and the memory controller M_CON, the volatile memory V_MEM, and the nonvolatile memory NV_MEM provided in the nonvolatile memory module NV_MM.
- FIGS. 3 and 5 are views depicting timing charts of signals exchanged between the memory access controller and the nonvolatile memory module in response to a normal read.
- FIGS. 3 and 5 depict a command bus CMD to which the memory access controller MAC outputs commands and so on, a read response signal RSP_R output by the nonvolatile memory module NV_MM, a data bus DQ, an ECC bus ECC, and a metadata signal META_DATA.
- the memory access controller MAC issues the normal read XREAD together with the address XADD thereof to the command bus CMD.
- a unique read identifier RID is attached to the normal read XREAD.
- the memory controller M_CON of the nonvolatile memory module accesses the volatile memory V_MEM, and when a cache hit is obtained such that a data response is possible, transmits a read-ready signal RD_RDY to the memory access controller MAC.
- the memory access controller MAC transmits a transmission request signal SEND to the nonvolatile memory module on the command bus CMD.
- the nonvolatile memory module After a predetermined time tSEND elapses following issuance of the transmission request signal SEND, the nonvolatile memory module transmits read data DATA to the memory access controller MAC. In addition to the read data DATA, the ECC signal ECC and the read identifier RID are transmitted. The nonvolatile memory module NV_MM, having performed a handshake with the memory access controller MAC through the read-ready signal RD_RDY and the transmission request signal SEND in this manner, then transmits the data to the memory access controller MAC.
- FIGS. 4 and 5 depict operations performed when the normal read is a cache miss.
- the memory access controller MAC issues the normal read XREAD together with the address XADD thereof to the command bus CMD.
- the unique read identifier RID is attached to the normal read XREAD.
- the memory controller M_CON of the nonvolatile memory module accesses the volatile memory V_MEM in response to the normal read XREAD and a cache miss is obtained
- the memory controller M_CON accesses the nonvolatile memory NV_MEM.
- Page unit data read from the nonvolatile memory NV_MEM are registered (stored) in the volatile memory, thereby enabling a data response, whereupon the read-ready signal RD_RDY is transmitted to the memory access controller MAC.
- the memory access controller MAC transmits the transmission request signal SEND to the nonvolatile memory module on the command bus CMD.
- the nonvolatile memory module After the predetermined time tSEND elapses following issuance of the transmission request signal SEND, the nonvolatile memory module transmits the read data DATA to the memory access controller MAC. In addition to the read data DATA, the ECC signal ECC and the read identifier RID are transmitted. Hence, when a cache miss occurs, the nonvolatile memory module NV_MM reads data from the nonvolatile memory and registers the read data in the volatile memory. Then, having performed a handshake with the memory access controller MAC through the read-ready signal RD_RDY and the transmission request signal SEND, the nonvolatile memory module NV_MM transmits the data to the memory access controller MAC.
- the latency with respect to the nonvolatile memory NV_MEM is longer than the latency with respect to the volatile memory, and therefore the latency up to data transmission by the nonvolatile memory module in case of a cache miss is considerably longer than that of a case where a cache hit is obtained.
- the latency is approximately 60 nsec, whereas in the case of a cache miss, the latency is on average 300 nsec.
- FIGS. 6 and 8 are views depicting an operation sequence executed in response to a speculative read.
- a speculative read command is denoted as SREAD, for example.
- FIGS. 7 and 9 are views depicting timing charts of signals exchanged between the memory access controller and the nonvolatile memory module in response to a speculative read.
- the memory access controller MAC issues the speculative read SREAD together with the address XADD thereof to the command bus CMD.
- the unique read identifier RID is likewise attached to the speculative read SREAD.
- the memory controller M_CON of the nonvolatile memory module accesses the volatile memory V_MEM and, regardless of whether a cache hit or a cache miss is obtained, issues a data response following the predetermined time tSEND from the speculative read SREAD.
- the nonvolatile memory module transmits data read from the volatile memory together with an ECC code and a cache hit flag H/M to the memory access controller MAC.
- the cache hit flag indicates a cache hit, the memory access controller acquires the received data.
- the nonvolatile memory module transmits dummy data together with an ECC code and a cache hit flag, and since the cache hit flag indicates a cache miss, the memory access controller discards the received data and waits for the read-ready signal.
- the nonvolatile memory module issues a data response following the predetermined time tSEND from the speculative read without performing a handshake using the read-ready signal RD_RDY and the transmission request signal SEND. Therefore, an equally short latency to that of a DRAM is realized, and as a result, a shorter latency, for example 30 nsec, than that of a case in which a cache hit is obtained in response to the normal read XREAD (60 nsec) is achieved.
- the memory access controller MAC issues the speculative read SREAD together with the address XADD thereof to the command bus CMD.
- the unique read identifier RID is attached to the speculative read SREAD.
- the memory controller M_CON of the nonvolatile memory module accesses the volatile memory V_MEM and, regardless of whether a cache hit or a cache miss is obtained, issues a data response following the predetermined time tSEND from the speculative read SREAD.
- the nonvolatile memory module transmits dummy read data together with an ECC code and the cache hit flag H/M to the memory access controller MAC. Since the cache hit flag indicates a cache miss, the memory access controller discards the received data and waits for the read-ready signal.
- the memory controller M_CON of the nonvolatile memory module accesses the nonvolatile memory, reads the data in a page area including the read address, and registers the read data in the volatile memory. Further, the nonvolatile memory module transmits the read-ready signal RD_RDY to the memory access controller MAC. In response to the read-ready signal RD_RDY, the memory access controller MAC transmits the transmission request signal SEND to the nonvolatile memory module on the command bus CMD.
- the nonvolatile memory module After the predetermined time tSEND elapses following issuance of the transmission request signal SEND, the nonvolatile memory module transmits the read data DATA to the memory access controller MAC. In addition to the read data DATA, the ECC signal ECC and the read identifier RID are transmitted.
- the speculative read is a cache miss
- a speculative data response is added following the predetermined time tSEND from the speculative read SREAD, and therefore the use efficiency of the bus between the memory access controller and the nonvolatile memory module deteriorates.
- the latency at an average 400 nsec, for example, is longer that of a case in which a cache miss is obtained in response to a normal read (30 nsec).
- FIG. 10 is a view of a table illustrating examples of the latency of the nonvolatile memory module.
- the latency of the normal read XREAD is 60 nsec in the case of a cache hit and 300 nsec on average in the case of a cache miss, for example.
- the latency of the speculative read SREAD is 30 nsec in the case of a cache hit and 400 nsec on average in the case of a cache miss, for example.
- the memory access controller MAC issues the speculative read SREAD in the case of a cache hit and the normal read XREAD in the case of a cache miss. Conversely, when the speculative read is issued in the case of a cache miss, the increased latency constitutes a large penalty.
- FIG. 11 is the nonvolatile memory module and a view depicting a relationship between the read data transmitted from the two memories thereof.
- FIG. 11 illustrates the respective sizes of the read data of the volatile memory V_MEM and the nonvolatile memory NV_MEM of the nonvolatile memory module NV_MM.
- each row of the matrix denotes a data size corresponding to one page, and the page on each row includes a plurality of bytes.
- one page includes 8 bytes, but in reality, the data on one page include several kilobytes of data.
- the number of pages in the nonvolatile memory NV_MEM is greater than the number of pages in the volatile memory V_MEM.
- the address of the read destination and the data size are attached to the read READ issued by the memory access controller MAC, and in the example depicted in FIG. 11 , the address ADD is B 300 and the data size is 64 bytes.
- the nonvolatile memory module In response to the read READ, when a cache hit is obtained in the volatile memory, the nonvolatile memory module reads 64 bytes of data with a single byte of data corresponding to the address B 300 at the beginning.
- the nonvolatile memory module reads one page of data having a page address Bxxx that includes the address B 300 from the nonvolatile memory NV_MEM, and then stores and registers the read data in the volatile memory V_MEM. Further, the nonvolatile memory module transmits 64 bytes of data, with a single byte of data corresponding to the address B 300 at the beginning, from the registered page of data to the memory access controller.
- the address B 300 is an address for a single byte of data, while the address Bxxx is a page address for a single page of data.
- the nonvolatile memory module when a cache miss is obtained in the volatile memory at the address B 300 of the read READ, the nonvolatile memory module reads one page of data including the read address B 300 from the nonvolatile memory and registers the read data in the volatile memory. Accordingly, when the address of a subsequently issued read READ is an address within the single page having the page address Bxxx, such as B 100 or B 200 , the nonvolatile memory module obtains a cache hit in the volatile memory and transmits data read from the volatile memory. Thus, the size (a byte unit) of the data read from the volatile memory is smaller than the size (a page unit) of data read from the nonvolatile memory. Therefore, when a cache miss occurs and a single page of the data in the nonvolatile memory is registered in the volatile memory, if the address of a subsequent read READ is on the same page, the likelihood of a cache hit is high.
- FIG. 12 is a view depicting an example configuration of the memory access controller MAC and the nonvolatile memory module NV_MM according to this embodiment.
- the memory access controller MAC receives a memory request M_RQ issued by the last level cache (the L2 cache in FIG. 1 ) of the processor, and stores the received memory request in a request queue RQ_QUE.
- a request issuance controller RQ_ISSU_CON selects an issuable memory request M_RQ from the request queue RQ_QUE. Whether or not a memory request is issuable is based on a read issuance rule such as whether or not a read issuance prohibition period has elapsed in the nonvolatile memory module, for example.
- a command issuance unit CMD_ISSU selects either the normal read XREAD or the speculative read SREAD and issues the selected read to the nonvolatile memory module NV_MM.
- the nonvolatile memory module NV_MM is configured similarly to that depicted in FIG. 1 .
- the nonvolatile memory module reads data relating to the received normal read XREAD or speculative read SREAD from the volatile memory or the nonvolatile memory in the nonvolatile memory module and transmits the read data to the memory access controller MAC via a data bus DATA. Together with the read data, the nonvolatile memory module transmits the error correction code ECC, the read identifier RID, and so on of the read data via a different bus.
- a data reception unit DATA_RCV of the memory access controller MAC receives the read data DATA, the ECC, and the RID, corrects an error in the data DATA on the basis of the ECC, and transmits the corrected data to the last level cache.
- FIG. 13 is a view depicting an example configuration of the command issuance unit CMD_ISSU in a memory access controller according to a first embodiment.
- the command issuance unit CMD_ISSU includes a page address table TBL on which a plurality of page addresses subject to the speculative read SREAD are registered, a pointer PTR for pointing to an entry in which the page address is to be updated, and a command selection unit CMD_SEL.
- the nonvolatile memory module stores the address information (the page address) corresponding to the data that have been cache-registered in the volatile memory in the cache tag and, in response to the normal read XREAD or the speculative read SREAD, performs a cache hit determination to determine whether or not the address information in the cache tag hits a read address.
- the number of entries for storing page addresses on the page address table TBL in the command issuance unit CMD_ISSU is much smaller than the number of entries in the cache tag.
- FIG. 14 is a view depicting a flowchart of an example operation of the command issuance unit CMD_ISSU.
- the command selection unit CMD_SEL selects either the normal read XREAD or the speculative read SREAD by referring to the page address table TBL, and issues the selected read to the nonvolatile memory module.
- the command selection unit CMD_SEL selects the normal read XREAD and issues the selected normal read to the memory module (S 4 ).
- a page address including the address of the read request is registered in an entry pointed to by the pointer PTR on the page address table TBL (S 4 ).
- the page address already registered in the entry is replaced with the new page address (S 4 ).
- the pointer PTR is incremented by +1.
- the pointer PTR is incremented in order of the index of the page address table (or in the reverse order), for example, every time a page address is updated.
- the command selection unit CMD_SEL selects and issues the speculative read SREAD (S 3 ).
- the reason for this is that the likelihood of a cache hit in the memory module is high.
- FIG. 15 is a view depicting a specific example of the operations performed by the command issuance unit in case 1 and case 2 .
- page addresses Axxx, Bxxx, Cxxx, and Dxxx are registered on the page address table TBL. This means that in the memory module, data having the page addresses Axxx, Bxxx, Cxxx, and Dxxx have been previously stored in the volatile memory.
- the address ADD of the read request is a byte address C 200 , which is included in the page address Cxxx on the page address table, and therefore a page address table hit HIT occurs. Accordingly, the command selection unit CMD_SEL issues the speculative read SREAD in response to the read request.
- the address ADD of the read request is a byte address E 030 , which is not included in any of the page addresses on the page address table, and therefore a page address table miss MISS occurs. Accordingly, the command selection unit CMD_SEL issues the normal read XREAD in response to the read request. Further, the command selection unit performs update processing UPDATE to replace the page address Axxx pointed to by the pointer PTR with a new page address Exxx that includes the address E 030 of the read request, whereby the pointer PTR is incremented by one.
- the memory access controller selects and issues the normal read XREAD and registers a page address including the address of the read request on the page address table.
- the nonvolatile memory module to which the normal read XREAD is issued accesses the nonvolatile memory when a cache miss in the cache tag occurs, reads the data of a single page having a page address that includes the read address, and registers the read data in the volatile memory.
- the command issuance unit CMD_ISSU of the memory access controller issues the speculative read SREAD in response to the aforesaid read request.
- the speculative read SREAD is highly likely to result in a cache hit in the nonvolatile memory module, and it is therefore likely that the nonvolatile memory module will be able to attach the hit flag to the response data in a hit state and transmit the response data thus to the memory access controller MAC without performing a handshake. As a result, it is likely that the memory access controller will be able to acquire the data with the minimum latency of the speculative read SREAD.
- the command selection unit CMD_SEL selects and issues the normal read XREAD. Even if the normal read XREAD results in a cache miss in the nonvolatile memory module, for example, the latency thereof is shorter than the latency produced by a cache miss in response to the speculative read SREAD, and therefore a situation in which data are received with the maximum latency produced by a cache miss in response to the speculative read SREAD can be avoided.
- the number of entries on the page address table in the memory access controller MAC is smaller than the number of entries of the cache tag provided in the nonvolatile memory module, and therefore the page address on the page address table does not always match the page address in the cache tag of the nonvolatile memory module.
- the normal read XREAD may also result in a cache hit
- the speculative read SREAD may also result in a cache miss.
- the processor that issues read requests to the nonvolatile memory module has a plurality of cores, as depicted in FIG. 1 , the plurality of cores issue read requests separately during the processes executed respectively thereby. It is therefore to be expected that the addresses of the read requests issued respectively by the plurality of cores will be random addresses. However, it is also generally foreseeable that the addresses of the read requests issued by a single core will exhibit temporal locality and spatial locality so as to be concentrated in a certain address area within a certain time period.
- the number of entries on the page address table provided in the memory access controller may be extremely small.
- the memory access controller MAC is built into the chip of the processor CPU, a fixed upper limit is set on the number of entries on the page address table.
- the addresses of the read requests from the multiple cores are random addresses among the plurality of cores.
- the number of entries on the page address table in the memory access controller is limited to a small number. Therefore, when the page address table is updated every time case 2 occurs, as in the first embodiment, a situation in which, before a read request having an address that is included in a page address newly registered on the page address table can be input again following registration, the page address is removed from the page address table occurs frequently.
- the number of entries in the cache tag provided in the nonvolatile memory module is large, and therefore the re-input read request results in a hit in the cache tag but a miss on the page address table. As a result, a likelihood that the speculative read SREAD can read with minimum latency becomes smaller.
- the page address table is updated at a lower frequency than the occurrence frequency of case 2 .
- the page address table is updated once each time the case 2 occurs a predetermined number of times.
- the page address table is updated once each time the case 1 occurs a predetermined number of times.
- the page address table is updated once each time a predetermined number of read requests are issued.
- FIG. 16 is a view depicting an example configuration of the command issuance unit CMD_ISSU provided in the memory access controller according to the second embodiment.
- the command issuance unit CMD_ISSU includes the page address table TBL on which a plurality of page addresses subject to the speculative read SREAD are registered, the pointer PTR for pointing to an entry in which the page address is to be updated, and the command selection unit CMD_SEL.
- the command issuance unit CMD_ISSU further includes a counter CNTR for counting the read requests from the cores and so on, and a register N storing a predetermined number of times N.
- the counter CNTR counts the number of times case 2 occurs, the number of times case 1 occurs, or the number of read requests.
- the page address table is updated with a new page address. Further, the predetermined number of times N in the register N may be rewritten (modified) by a user or the like.
- FIG. 17 is a view depicting a flowchart of an example operation of the command issuance unit CMD_ISSU.
- the counter CNTR counts the number of read requests.
- the command selection unit CMD_SEL selects either the normal read XREAD or the speculative read SREAD by referring to the page address table TBL, and issues the selected read to the nonvolatile memory module.
- the command selection unit CMD_SEL determines whether the address of the input read request corresponds to case 1 (a hit), in which the address is included in one of the page addresses registered on the page address table, or case 2 (a miss), in which the address is not included in any of the page addresses (S 2 ).
- the command selection unit CMD_SEL selects and issues the normal read XREAD and increments the counter by +1 (S 15 ).
- the command selection unit CMD_SEL selects and issues the normal read XREAD, replaces or updates the entry pointed to by the pointer on the page address table with a page address that includes the address of the read request, and resets the counter to 0 (S 16 ).
- FIG. 18 is a view depicting specific examples of operations performed by the command issuance unit in a plurality of cases.
- the page addresses Axxx, Bxxx, Cxxx, and Dxxx are registered on the page address table TBL.
- the memory request M_RQ is a read request
- the address ADD of the read request is the byte address C 200 , which is included in the page address Cxxx on the page address table. Therefore, the address ADD is a hit HIT. Accordingly, the command selection unit CMD_SEL issues the speculative read SREAD. Further, since the counter has not yet reached N, the command selection unit increments the counter by +1 (S 12 ).
- the memory request is a read request
- the address ADD of the read request is the byte address E 030 , which is not included in any of the page addresses on the page address table.
- the address ADD is a miss MISS. Accordingly, the command selection unit CMD_SEL issues the normal read XREAD. Further, since the counter has not yet reached N, the command selection unit increments the counter by +1 (S 15 ). Note, however, that since the counter has not yet reached N, the command selection unit does not update the page address table with a new page address.
- the address ADD of the memory request is a byte address D 300 , which is included in the page address Dxxx on the page address table, and therefore the address ADD is a hit HIT. Accordingly, the command selection unit CMD_SEL issues the speculative read SREAD. Further, the counter has reached N, and therefore the command selection unit does not increment the counter (S 13 ).
- the address ADD of the memory request is a byte address F 300 , which is not included in any of the page addresses on the page address table, and therefore the address ADD is a miss MISS. Accordingly, the command selection unit CMD_SEL issues the normal read XREAD (S 16 ). Further, since the counter has reached N, the command selection unit CMD_SEL performs the update processing UPDATE to replace the page address Axxx pointed to by the pointer PTR with a new page address Fxxx that includes the address F 300 of the read request, and resets the pointer PTR to 0 (S 16 ).
- FIG. 19 is a view depicting an example operation in a case where the memory access controller updates the page address table every time a miss occurs.
- the address of the read request and the issued normal read XREAD or speculative read SREAD are depicted on the left side along a temporal axis t extending in a vertical direction, and example transitions of the page address table corresponding to times 0 - 16 are depicted on the right side.
- the page address table includes 4 entries.
- the read request addresses on the left side are written as 4-bit hexadecimals, and on a page address table P_ADD_TBL on the right side, page addresses in which the most significant bit of the read request address is displayed as the most significant bit and the 3 lower order bits are displayed as “x” are written in the entries of indices 0 to 3 pointed to by a pointer P.
- the read request addresses extending along the temporal axis t on the left side are examples from a multicore processor and are therefore random.
- the page address table on the right side is updated with a new page address at each of times 4 , 5 , 6 , 7 , 8 , 10 , 11 , 12 , 13 , 14 , 15 , and 16 .
- an address “5F40” of a read request issued at time 9 is a hit in a page address “5xxx” registered on the page address table at time 5 , and therefore the speculative read SREAD is issued at time 9 .
- no page address including the address of the read request is registered on the table, and therefore a miss is obtained and the normal read XREAD is issued. This is because the page address table is updated every time a miss occurs.
- FIG. 20 is a view depicting an example operation in which the memory access controller updates the page address table once every N read requests.
- the example addresses on the temporal axis on the left side are the same as those of FIG. 19 .
- the page address table on the right side is not updated with new page addresses from times 0 - 15 , but a page address update is performed at time 16 .
- the speculative read SREAD is issued at times 8 , 10 , 12 , and 13 .
- the speculative read SREAD can be issued appropriately even when the number of entries on the page entry table is small and read requests having random read addresses are input. As a result, the frequency with which data are acquired with the short latency produced by a cache hit in response to the speculative read can be increased.
- the predetermined number of times N indicating the update frequency of the page address table is preferably set at an optimum value in consideration of various factors. For example, (1) the predetermined number of times N is larger as the ratio of the number of entries in the cache tag in the memory module to the number of entries in the page address table is smaller, leading to a reduction in the frequency with which a page address registered on the page address table is replaced with a page address that includes the address of the read request, (2) the predetermined number of times N is larger as the number of cores that issue read requests to the memory module is larger, leading to a reduction in the frequency with which a registered page address is replaced with a page address that includes the address of the read request, and (3) by causing the processor to execute a predetermined program while modifying the predetermined number of times N, the predetermined number of times N is set so as to minimize the average latency. In the above (1), it is preferable that the replaced frequency of the page address table is lower, when the number of entries in the page address table is smaller.
- FIG. 21 is a view depicting a comparison between a case in which either only the normal read XREAD is issued or only the speculative read SREAD is issued and a case such as that of the second embodiment, in which either XREAD or SREAD is issued depending on whether a hit occurs on the page address table.
- the horizontal axis depicts the hit rate in the cache tag of the volatile memory V_MEM and the vertical axis depicts the average latency.
- FIG. 21 illustrates the results of a simulation in which read request addresses were generated at random, as in FIGS. 19 and 20 .
- the normal read XREAD is issued appropriately with steadily greater frequency as the cache hit rate decreases, and as a result, the difference in latency between the cases increases.
- the speculative read SREAD is issued appropriately with steadily greater frequency as the cache hit rate increases, and as a result, the difference in latency between the cases increases.
- the average latency of the second embodiment is shorter than that of the comparison subject.
- normal reads and speculative reads are selected and issued appropriately in response to read requests issued to a nonvolatile memory module that mounts a volatile memory and a nonvolatile memory, and as a result, the average latency can be shortened.
- normal reads and speculative reads can be selected appropriately, and as a result, the latency up to the acquisition of read data can be shortened.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Human Computer Interaction (AREA)
- Computational Mathematics (AREA)
- Mathematical Analysis (AREA)
- Mathematical Optimization (AREA)
- Pure & Applied Mathematics (AREA)
- Memory System Of A Hierarchy Structure (AREA)
- Memory System (AREA)
Abstract
Description
Claims (11)
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2018-239503 | 2018-12-21 | ||
JP2018239503A JP7140972B2 (en) | 2018-12-21 | 2018-12-21 | Arithmetic processing device, information processing device, and method of controlling arithmetic processing device |
JPJP2018-239503 | 2018-12-21 |
Publications (2)
Publication Number | Publication Date |
---|---|
US20200201556A1 US20200201556A1 (en) | 2020-06-25 |
US11314438B2 true US11314438B2 (en) | 2022-04-26 |
Family
ID=71097661
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/695,282 Active 2040-09-11 US11314438B2 (en) | 2018-12-21 | 2019-11-26 | Arithmetic processing device, information processing device, and control method for arithmetic processing device |
Country Status (2)
Country | Link |
---|---|
US (1) | US11314438B2 (en) |
JP (1) | JP7140972B2 (en) |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5742792A (en) * | 1993-04-23 | 1998-04-21 | Emc Corporation | Remote data mirroring |
US6249851B1 (en) * | 1998-08-25 | 2001-06-19 | Stmicroelectronics, Inc. | Computer system having non-blocking cache and pipelined bus interface unit |
US20030097531A1 (en) * | 2001-10-16 | 2003-05-22 | International Business Machines Corp. | Dynamic hardware and software performance optimizations for super-coherent SMP systems |
US20040039880A1 (en) * | 2002-08-23 | 2004-02-26 | Vladimir Pentkovski | Method and apparatus for shared cache coherency for a chip multiprocessor or multiprocessor system |
US20060143406A1 (en) * | 2004-12-27 | 2006-06-29 | Chrysos George Z | Predictive early write-back of owned cache blocks in a shared memory computer system |
US20060156155A1 (en) * | 2004-12-29 | 2006-07-13 | Sudhanva Gurumurthi | Detecting errors in directory entries |
US20170357604A1 (en) | 2016-06-08 | 2017-12-14 | Samsung Electronics Co., Ltd. | System and method for operating a drr-compatible asynchronous memory module |
Family Cites Families (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP3175675B2 (en) * | 1997-12-04 | 2001-06-11 | 日本電気株式会社 | Prefetch control device |
JP3969009B2 (en) | 2001-03-29 | 2007-08-29 | 株式会社日立製作所 | Hardware prefetch system |
US10482043B2 (en) | 2016-08-29 | 2019-11-19 | Advanced Micro Devices, Inc. | Nondeterministic memory access requests to non-volatile memory |
US10866897B2 (en) | 2016-09-26 | 2020-12-15 | Samsung Electronics Co., Ltd. | Byte-addressable flash-based memory module with prefetch mode that is adjusted based on feedback from prefetch accuracy that is calculated by comparing first decoded address and second decoded address, where the first decoded address is sent to memory controller, and the second decoded address is sent to prefetch buffer |
US10127074B2 (en) | 2017-01-27 | 2018-11-13 | Futurewei Technologies, Inc. | Transaction identification synchronization |
-
2018
- 2018-12-21 JP JP2018239503A patent/JP7140972B2/en active Active
-
2019
- 2019-11-26 US US16/695,282 patent/US11314438B2/en active Active
Patent Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5742792A (en) * | 1993-04-23 | 1998-04-21 | Emc Corporation | Remote data mirroring |
US6249851B1 (en) * | 1998-08-25 | 2001-06-19 | Stmicroelectronics, Inc. | Computer system having non-blocking cache and pipelined bus interface unit |
US20030097531A1 (en) * | 2001-10-16 | 2003-05-22 | International Business Machines Corp. | Dynamic hardware and software performance optimizations for super-coherent SMP systems |
US20040039880A1 (en) * | 2002-08-23 | 2004-02-26 | Vladimir Pentkovski | Method and apparatus for shared cache coherency for a chip multiprocessor or multiprocessor system |
US20060143406A1 (en) * | 2004-12-27 | 2006-06-29 | Chrysos George Z | Predictive early write-back of owned cache blocks in a shared memory computer system |
US20060156155A1 (en) * | 2004-12-29 | 2006-07-13 | Sudhanva Gurumurthi | Detecting errors in directory entries |
US20170357604A1 (en) | 2016-06-08 | 2017-12-14 | Samsung Electronics Co., Ltd. | System and method for operating a drr-compatible asynchronous memory module |
JP2017220237A (en) | 2016-06-08 | 2017-12-14 | 三星電子株式会社Samsung Electronics Co.,Ltd. | Memory module, system including the same, and method for operating the same |
Also Published As
Publication number | Publication date |
---|---|
JP2020101983A (en) | 2020-07-02 |
JP7140972B2 (en) | 2022-09-22 |
US20200201556A1 (en) | 2020-06-25 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9836403B2 (en) | Dynamic cache allocation policy adaptation in a data processing apparatus | |
US9785545B2 (en) | Method and apparatus for providing dual memory access to non-volatile memory | |
US11379381B2 (en) | Main memory device having heterogeneous memories, computer system including the same, and data management method thereof | |
JP6599898B2 (en) | Providing memory bandwidth compression using a compression memory controller (CMC) in a system with a central processing unit (CPU) | |
US11487616B2 (en) | Write control for read-modify-write operations in cache memory | |
US11210020B2 (en) | Methods and systems for accessing a memory | |
KR20190096801A (en) | System and method for nvme inter command association in ssd storage | |
US9135177B2 (en) | Scheme to escalate requests with address conflicts | |
US8180965B2 (en) | System and method for cache access prediction | |
US20150143045A1 (en) | Cache control apparatus and method | |
US11169924B2 (en) | Prefetch management in a hierarchical cache system | |
US11977491B2 (en) | Prefetch kill and revival in an instruction cache | |
US11836092B2 (en) | Non-volatile storage controller with partial logical-to-physical (L2P) address translation table | |
US11314438B2 (en) | Arithmetic processing device, information processing device, and control method for arithmetic processing device | |
US20190012279A1 (en) | Computer system, communication device, and storage control method | |
EP4124963A1 (en) | System, apparatus and methods for handling consistent memory transactions according to a cxl protocol | |
US10789001B1 (en) | Posted operation data control | |
US20140136796A1 (en) | Arithmetic processing device and method for controlling the same | |
US11734005B2 (en) | Processor with split read | |
US11216379B2 (en) | Fast cache loading with zero fill | |
US11726920B2 (en) | Tag processing for external caches | |
JP6665429B2 (en) | Arithmetic processing device, information processing device, and control method for information processing device | |
CN114090080A (en) | Instruction cache, instruction reading method and electronic equipment | |
KR20150009883A (en) | Cache memory system and operating method for the same |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: FUJITSU LIMITED, JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:IWASAKI, SHINICHI;REEL/FRAME:051139/0878 Effective date: 20191101 |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: AWAITING TC RESP., ISSUE FEE NOT PAID |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |