KR102340444B1 - A GPU cache bypassing method and apparatus with the adoption of monolithic 3D based network-on-chip - Google Patents

Abstract

The present invention discloses a GPU cache bypass method and apparatus utilizing a NoC structure based on a monolithic 3D integration technology. According to the present embodiment, there is provided a type classification structure for tracking the L1 cache utilization rate of an individual warp, classifying the type of the individual warp, and determining a bypass threshold value of the individual warp according to the classified warp type; A request bypass for adaptively determining a bypass for a memory request of each warp by comparing a saturation counter (SC) that records the bypass history of the individual warp and a threshold value of the bypass of the individual warp A GPU cache bypass device using a monolithic 3D integration technology-based NoC structure including a monolithic 3D-based NoC (Network on Chip) to which a memory request bypassed by the pass structure and the request bypass structure is input provided

Description

A GPU cache bypassing method and apparatus with the adoption of monolithic 3D based network-on-chip }

The present invention relates to a GPU cache bypass method and apparatus utilizing a NoC structure based on a monolithic 3D integration technology.

Graphics Processing Units (GPUs) have been used to improve the performance of general-purpose applications over the past decade. A large number of threads (warps) can be started simultaneously in order to utilize the computational resources of the GPU. However, massively multi-threading does not always lead to performance benefits.

In particular, cache contention often occurs as the cache hit ratio becomes very low because too many threads share the limited capacity of the L1 data cache (L1 cache). Also, mishandled resources, such as miss status holding registers (MSHR) and miss buffers, often cause congestion. As a result, the efficiency of the cache layer on the GPU is reduced by the L1 cache bottleneck.

Due to the limited L1 cache capacity, traditional cache replacement policies are not always effective in resolving cache contention. For applications with poor data locality, using an L1 cache can cause poor performance due to congested resources. In this context, allowing memory requests to bypass the cache is considered a solution to improve GPU cache management.

In GPUs, the L1 cache is connected to the lower level memory (eg L2 cache) by a Network on Chip (NoC). When an L1 cache miss occurs, the memory request must go through the NoC to retrieve the data. If the network delay is greater than the threshold or if the network buffer of incoming packets is unavailable, the NoC is congested. When cache bypass is on, the L1 cache loses its role of filtering memory requests to lower levels of memory. Therefore, more memory requests are directed to the NoC for a shorter period of time, exacerbating NoC congestion. Addressing NoC congestion caused by cache bypass requires NoCs to provide better network latency and throughput. .

3D integration can reduce interconnect latency by stacking multiple layers.

The most commonly used 3D integration is based on TSV (Through-Silicon Via) to realize vertical connectivity. However, TSV-based 3D integration still has major drawbacks such as large diameter and pitch of TSV, which increases the delay and area of other logic.

Patent Registration Publication No. 10-1761301

In order to solve the problems of the prior art, the present invention intends to propose a GPU cache bypass method and apparatus using a NoC structure based on a monolithic 3D integration technology capable of reducing a delay in case of a cache miss.

In order to achieve the above object, according to an embodiment of the present invention, the L1 cache utilization rate of an individual warp is tracked, the type of the individual warp is classified, and the individual warp is bypassed according to the classified warp type. a type classification structure that determines the threshold of ; A request bypass for adaptively determining a bypass for a memory request of each warp by comparing a saturation counter (SC) that records the bypass history of the individual warp and a threshold value of the bypass of the individual warp A GPU cache bypass device using a monolithic 3D integration technology-based NoC structure including a monolithic 3D-based NoC (Network on Chip) to which a memory request bypassed by the pass structure and the request bypass structure is input provided

The type classification structure may include: a tracking unit for tracking an L1 cache utilization rate of the individual warps; a word type determining unit that determines at least three types according to the L1 cache utilization ratio of the individual warps; and a bypass threshold allocator for allocating a bypass threshold of the individual warp according to the determined type.

The L1 cache utilization ratio is the number of cache accesses (a), the number of cache misses (m), and the number of reservation failures (r) of the individual warps, and the type determining unit includes an average of m/a and r/a of all warps and the By comparing the m/a and r/a of the individual warps, the type of the individual warps can be determined.

The type determining unit may determine a T1 warp for a warp in which m/a and r/a are greater than or equal to the average of all warps, a T2 warp for a warp in which one of m/a or r/a is greater than the average of all warps, m/ A warp in which a and r/a are smaller than the average of all warps may be determined as a T3 warp.

The bypass threshold allocator may allocate different bypass thresholds to the T1, T2, and T3 warps.

When the probe result of the memory request is a tag miss, the value of the saturation counter may increase.

The saturation counter is indexed for each ID of the individual warp, and the request bypass structure may determine to bypass the L1 cache only when the value of the saturation counter is equal to or greater than a bypass threshold value of the corresponding warp.

The monolithic 3D via-based NoC includes a plurality of routers having a grid-type mesh structure, each router has a unique address for location definition in the network, and has four basic directions (North, East, South and West). ) and 6 physical ports in the up/down direction.

According to another aspect of the present invention, in the type classification structure, tracking the L1 cache utilization rate of an individual warp, classifying the type of the individual warp, and determining a threshold value of the bypass of the individual warp according to the classified warp type step; And in the request bypass structure, the bypass for the memory request of the individual warp is determined by comparing a saturation counter (SC) that records the bypass history of the individual warp and a threshold value of the bypass of the individual warp. A GPU using a monolithic 3D integration technology-based NoC structure, comprising the step of adaptively determining, wherein the memory request bypassed by the request bypass structure is input to a monolithic 3D-based NoC (Network on Chip) A cache bypass method is provided.

According to the present invention, there is an advantage that more efficient cache management is possible by alleviating L1 cache contention and NoC congestion.

1 is a diagram illustrating a GPU cache bypass device according to an embodiment of the present invention.
2 is a diagram illustrating a warp classification structure according to an embodiment of the present invention.
3 is a diagram illustrating a request bypass structure according to an embodiment of the present invention.
4 is a diagram illustrating a monolithic 3D-based NoC structure according to an embodiment of the present invention.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail.

However, this is not intended to limit the present invention to specific embodiments, and it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention.

1 is a diagram illustrating a GPU cache bypass device according to an embodiment of the present invention.

As shown in Fig. 1, the GPU cache bypass apparatus according to the present embodiment includes a warp classification structure (WCS, 100), a request bypass structure (RBS, 102), and a monolithic 3D structure. It may include an underlying NoC 104 .

The warp classification structure 100 tracks the L1 cache utilization rate of each warp, classifies the type of each warp, and determines a bypass threshold according to the classified warp type.

2 is a diagram illustrating a warp classification structure according to an embodiment of the present invention.

A warp is the smallest instruction processing unit, also called a thread.

The warp classification structure 100 according to the present embodiment allocates a high bypass probability to a memory request in a warp having a low L1 cache utilization rate and a low bypass probability to a memory request in a warp having a high L1 cache utilization ratio.

Referring to FIG. 2 , the warp classification structure 100 may include a tracking unit 200 , a warp type determiner 202 , and a bypass threshold allocator 204 .

The warp classification structure 100 is designed to classify each warp into different types according to the L1 cache utilization rate of each warp.

To estimate the L1 cache utilization of each warp, the L1 cache is accessed during a period called the training period at the start of kernel execution.

During the training period, the tracking unit 200 tracks the number of cache accesses per warp (a), the number of cache misses (m), and the number of reservation failures (r).

For L1 cache miss, the number of reservation failures is counted when the miss processing resource (eg MSHR, miss buffer) is not available.

When the training period is over, the warp type determining unit 202 determines the plurality of warps as one of a T1 warp (high bypass probability), a T2 warp (normal bypass probability), and a T3 warp (low bypass probability).

A T1 warp is a warp in which m/a and r/a are greater than or equal to the mean of all warps, a T2 warp is a warp in which either m/a or r/a is greater than the mean of all warps, and a T3 warp is a warp in which m/a or r/a is greater than the mean of all warps. a and r/a are warps smaller than average.

The bypass threshold allocator 204 assigns a bypass threshold Θ to each warp type, and assigns a T1 warp a higher threshold than a T3 warp.

3 is a diagram illustrating a request bypass structure according to an embodiment of the present invention.

The request bypass structure 102 includes a first identifier 300 that identifies a warp waiting in a request queue, a 2-bit saturation counter that records the bypass history of each warp (Saturating Counter, SC, 302); It may include a second identifier 304 and a bypass determiner 306 .

The request bypass structure 102 adaptively determines a bypass for a memory request by using a 2-bit Saturating Counter (SC) that records the bypass history of the warp.

The saturation counter 302 is indexed by warp ID.

The warp type and bypass threshold for each warp ID are determined by the warp classification structure 100 .

The request bypass structure 102 is operational after the training period. When a memory request is entered through the request queue, it immediately accesses the L1 cache tag array, which is called a tag probe.

The probe result of the memory request is used to update the corresponding saturation counter 302 in the request bypass structure 102 . The initial value of all saturation counters is '0'. If the probe result is a tag miss, that is, there is no matching tag, the value of the saturation counter is incremented. Otherwise, the value of the saturation counter decreases.

The bypass determining unit 306 compares the value of the saturation counter 302 with the bypass threshold Θ of the warp that issues the memory request, and only when the value of the saturation counter 302 is greater than or equal to the bypass threshold. It decides to bypass the L1 cache.

In this case, the returned data is sent directly to the single-instruction, multiple-thread (SIMT) core rather than the L1 cache. Otherwise, the memory request accesses the L1 cache.

4 is a diagram illustrating a monolithic 3D-based NoC structure according to an embodiment of the present invention.

4 shows the structure of an M3D network-on-chip (M3D NoC) composed of general components such as routers and links, and memory requests bypassed by the request bypass structure 102 of FIG. 104) is entered.

The router of the M3D NoC 104 according to the present embodiment has a grid-type mesh structure and is arranged in several layers.

Each router interfaces with a Processing Element (PE) through a Network Interface Controller (NIC). A router has a unique address (X, Y, Z) to define its location on the network. In addition to the default directions (North, East, South and West), two physical ports (one Up, the other Down) are added to the router with their associated buffers and VC/Switch arbiters.

The router's crossbar expands from 5x5 to 7x7. A router on one layer is connected through a 2D wire, and a router on the other layer is connected through a monolithic inter-via (MIV). M3D NoC uses XYZ routing algorithm.

The above-described embodiments of the present invention have been disclosed for purposes of illustration, and various modifications, changes, and additions will be possible within the spirit and scope of the present invention by those skilled in the art having ordinary knowledge of the present invention, and such modifications, changes and additions should be regarded as belonging to the following claims.

Claims (9)

a type classification structure for tracking an L1 cache utilization rate of an individual warp, classifying a type of the individual warp, and determining a threshold value of a bypass of the individual warp according to the classified warp type;
A request bypass for adaptively determining a bypass for a memory request of the individual warp by comparing a saturation counter (SC) that records the bypass history of the individual warp and a threshold value of the bypass of the individual warp path structure and
Including a monolithic 3D-based NoC (Network on Chip) to which a memory request bypassed by the request bypass structure is input,
A GPU cache bypass device using a monolithic 3D integration technology-based NoC structure in which the value of the saturation counter is increased when the probe result of the memory request is a tag miss. According to claim 1,
The type classification structure is
a tracking unit that tracks the L1 cache utilization rate of the individual warps;
a word type determination unit that determines at least three types according to the L1 cache utilization rate of the individual warps; and
A GPU cache bypass device utilizing a NoC structure based on a monolithic 3D integration technology, comprising a bypass threshold allocator for allocating a bypass threshold value of the individual warp according to the determined type. 3. The method of claim 2,
The L1 cache utilization ratio is the number of cache accesses (a), the number of cache misses (m), and the number of reservation failures (r) of the individual warps,
The type determining unit compares the average of m/a and r/a of all warps with m/a and r/a of the individual warps to determine the type of the individual warp, a NoC structure based on a monolithic 3D integration technology Utilized GPU cache bypass device. 4. The method of claim 3,
The type determining unit may determine a T1 warp for a warp in which m/a and r/a are greater than or equal to the average of all warps, a T2 warp for a warp in which one of m/a or r/a is greater than the average of all warps, m/ GPU cache bypass device utilizing a NoC structure based on monolithic 3D integration technology that determines a warp with a and r/a less than the average of all warps as a T3 warp. 5. The method of claim 4,
The bypass threshold allocator is a GPU cache bypass device utilizing a monolithic 3D integration technology-based NoC structure for allocating different bypass thresholds to the T1, T2, and T3 warps. delete According to claim 1,
The saturation counter is indexed by ID of the individual warp,
The request bypass structure is a GPU cache bypass device using a monolithic 3D integration technology-based NoC structure that determines to bypass the L1 cache only when the value of the saturation counter is equal to or greater than the bypass threshold of the corresponding warp. According to claim 1,
The monolithic 3D-based NoC includes a plurality of routers having a grid-type mesh structure,
Each router has a unique address for defining its location on the network, and a NoC based on monolithic 3D aggregation technology that includes 4 primary directions (North, East, South and West) and 6 physical ports in up/down directions. GPU cache bypass device utilizing architecture. in the type classification structure, tracking the L1 cache utilization rate of each warp, classifying the type of the individual warp, and determining a threshold value of bypass of the individual warp according to the classified warp type; and
In the request bypass structure, the bypass for the memory request of the individual warp is adapted by comparing a saturation counter (SC) that records the bypass history of the individual warp and a threshold value of the bypass of the individual warp. including the step of making a negative decision,
The memory request bypassed by the request bypass structure is input to a monolithic 3D-based NoC (Network on Chip),
A GPU cache bypass method using a monolithic 3D integration technology-based NoC structure in which the value of the saturation counter is increased when the probe result of the memory request is a tag miss.

Images