RU2775818C2 - Cache-based trace recording using data of cache coherence protocol - Google Patents

Abstract

FIELD: computer technology.

SUBSTANCE: it is determined that at least following conditions are fulfilled: (i) the operation has caused the interaction between a specific cache row from the mentioned set of cache rows and the mentioned one or more reserve storages; (ii) the registration is allowed for a specific processing unit from the mentioned set of processing units, which has caused this operation; (iii) the mentioned specific cache row is a registration participant; and (iv) CCP indicates that there is data that should be registered in the trace, based on the mentioned operation; and, based on at least the determination of that the mentioned conditions are fulfilled, data is registered in the trace, wherein data can be used for the reproduction of the mentioned operation.

EFFECT: reduction in a trace file size by several orders and, thereby, significant reduction in the cost of trace recording.

20 cl, 19 dwg, 1 tbl

Description

BACKGROUND OF THE INVENTION

[0001] When writing code during the development of software applications, developers typically spend a significant amount of time "debugging" the code to look for run-time errors and other errors in the source code. In doing so, developers can take several approaches to repeating and localizing a defect in the source code, such as observing the behavior of the program based on different inputs, inserting debug code (for example, to print the values of variables to track execution branches, etc.), temporary removal of parts of the code, etc. Tracking down run-time errors to identify code defects can take up a significant amount of application development time.

[0002] Many types of debugging applications ("debuggers") have been developed to assist developers in the process of debugging code. These tools offer developers the ability to trace, visualize, and modify the execution of native code. For example, debuggers may visualize the execution of instructions presented in the code, may reflect the values of variables in the code at different times during code execution, may allow developers to change code execution paths, and/or may allow developers to set "breakpoints" and/or "watchpoints" in code elements of interest (which, when reached during execution, cause code execution to pause), among other things.

[0003] The perspective form of debugging applications allows "time-traversing", "reverse", or "chronological" debugging to be performed. In "time-advance" debugging, the execution of a program (eg, executable objects such as threads of execution) is recorded/traced by a trace application into one or more trace files. These trace file(s) can then be used to replay program execution later, either for forward or backward analysis. For example, "jump-in-time" debuggers can allow the developer to set break/watch points in forward order (like regular debuggers) and break/watch points in reverse order.

SUMMARY OF THE INVENTION

[0004] The embodiments herein enhance "time-advance" debug logging by using the processor's shared cache, along with its cache coherence protocol (CCP), to determine what data should be logged (logged). ) in the trace file. This can reduce the size of the trace file by several orders of magnitude compared to previous approaches, thereby significantly reducing the overhead of recording a trace.

[0005] In some cases, embodiments are implemented in computing environments that include (i) a plurality of processing units and (ii) a cache memory containing a plurality of cache lines that are used to cache data from one or more backing stores and that are shared (shared) by multiple processing units. The consistency of data across multiple cache lines and one or more backing stores is governed by a cache coherence protocol.

[0006] These embodiments include performing a cache-based trace entry using CCP data. These embodiments include determining that an operation has caused an interaction between a particular cache line of the plurality of cache lines and one or more backing stores, that registration is allowed for a particular processing unit of the plurality of processing units that caused the operation, that the particular cache line is a member logging, and that the CCP indicates that there is data to be logged in the trace. Based on at least these definitions, the embodiments cause this data to be logged in the trace. The data can be used to reproduce the operation.

[0007] This Summary is provided to introduce in a simplified form a selection of concepts that are further described below in the Detailed Description of the Invention. This Summary is not intended to identify key features or essential features of the claimed invention, nor is it intended to be used as an aid in determining the scope of the claimed invention.

BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to describe how the above and other advantages and features of the present invention can be obtained, a more detailed description of the present invention, briefly described above, will be provided with reference to its separate embodiments, which are illustrated in the accompanying drawings. With the understanding that these drawings depict only exemplary embodiments of the present invention and, therefore, should not be construed as limiting the scope of the present invention, the present invention will be described and explained in further specificity and detail through the use of the accompanying drawings, in which:

[0009] FIG. 1 shows an exemplary computing environment that facilitates "bit-accurate" recording of code execution trace across shared caches using Cache Coherence Protocol (CCP) data;

[0010] FIG. 2 shows an example of a shared cache;

[0011] FIG. 3 shows a flowchart of an exemplary method for performing a cache-based trace entry using CCP data;

[0012] FIG. 4A shows an exemplary shared cache that extends each of its cache lines with one or more additional accounting bits;

[0013] FIG. 4B shows an example of a shared cache that includes one or more reserved cache lines for storing accounting bits applicable to regular cache lines;

[0014] FIG. 5 shows an example of associative cache mappings;

[0015] FIG. 6A shows a table that shows exemplary read and write operations by four processing units on a single line in a shared cache;

[0016] FIG. 6B shows a table that demonstrates an exemplary watch cache coherence state based on the read and write actions shown in FIG. 6A;

[0017] FIG. 6C shows a table that shows exemplary data stored in accounting bits (ie, block bits, index bits, and/or flag bits) of a shared cache based on the read and write actions shown in FIG. 6A;

[0018] FIG. 6D shows a table that shows exemplary logging data that may be written to trace files in connection with the read and write operations shown in FIG. 6A;

[0019] FIG. 7A shows an example where some read->read transitions may be excluded from the trace depending on how processors are tracked;

[0020] FIG. 7B shows an example of data logging that skips the read->read transitions highlighted in FIG. 7A;

[0021] FIG. 7C shows a table that shows the logging of data that can be written if "index bits" are used and the indices are updated on reads;

[0022] FIG. 8A shows an exemplary computing environment including two processors, each including four processing units, and L1-L3 caches;

[0023] FIG. 8B shows a table that shows exemplary read and write operations performed by some of the processing units shown in FIG. 8A;

[0024] FIG. 9A shows a table that shows exemplary reads and writes by two processing units;

[0025] FIG. 9B shows an example of a table that compares how registration entries can be contributed by an environment that provides CCP block information plus a cache line flag bit, as opposed to an environment that provides CCP index information plus a cache line flag bit;

[0026] FIG. 10A shows an example of different parts of a memory address and their relationship to associative caches; and

[0027] FIG. 10B shows an example of logging cache misses and cache evictions in an associative cache.

DETAILED DESCRIPTION OF THE INVENTION

[0028] The embodiments herein enhance "time-advance" debug records by using the processor's shared cache, along with its cache coherency protocol, to determine what data should be logged in the trace file. This can reduce the size of the trace file by several orders of magnitude compared to previous approaches, thereby significantly reducing the overhead of recording a trace.

[0029] FIG. 1 shows an exemplary computing environment 100 that facilitates "bit-accurate" recording of code execution traces across shared caches using Cache Coherence Protocol data. As depicted, embodiments may comprise or utilize a computer system 101, whether specialized or general purpose, that includes computer hardware such as, for example, one or more processors 102, system memory 103, one or more data stores 104, and/ or I/O hardware 105.

[0030] Embodiments within the scope of the present invention include physical and other computer-readable media for carrying or storing computer-executable instructions and/or data structures. Such computer-readable media can be any suitable media that can be accessed by computer system 101. Computer-readable media that stores computer-executable instructions and/or data structures are computer storage devices. Computer-readable media that carry computer-executable instructions and/or data structures are data transfer media. Thus, by way of example, and not limitation, embodiments of the present invention may comprise at least two distinct types of computer-readable media: computer storage devices and communication media.

[0031] Computer storage devices are physical hardware devices that store computer-executable instructions and/or data structures. Computer storage devices include various computer hardware such as RAM, ROM, EEPROM, solid state drives (SSD), flash memory, phase change memory (PCM), optical disk storage, magnetic disk storage, or other magnetic devices. storage device(s), or any other hardware device(s) that can be used to store program code in the form of computer executable instructions or data structures, and that can be accessed and executed by computer system 101 to implement the disclosed functionality of the present invention. Thus, for example, computer storage devices may include the depicted system memory 103, the depicted data store 104, which may store computer-executable instructions and/or data structures, or other storage, such as storage built into the processing device as will be discussed below. .

[0032] The transmission media may include a network and/or data channels that can be used to carry program code in the form of computer executable instructions or data structures and that can be accessed by computer system 101. A "network" is defined as one or a plurality of data transmission channels that enable electronic data to be transported between computer systems and/or modules and/or other electronic devices. When information is moved or provided over a network or other communications connection (whether wired, wireless, or a combination of wired or wireless) to a computer system, the computer system may consider the connection as a medium of transmission. Combinations of the above should also be included within the scope of computer readable media. For example, I/O hardware 105 may include hardware (eg, a network interface module (eg, "NIC")) that establishes a connection to a network and/or data link that can be used to carry program code in computer executable form. instructions or data structures.

[0033] In addition, when reaching various components of a computer system, program code in the form of computer-executable instructions or data structures can automatically move from transmission media to computer storage devices (or vice versa). For example, computer-executable instructions or data structures received over a network or data link may be buffered in RAM within the NIC (eg, I/O hardware 105) and subsequently moved to system memory 103 and/or less volatile computer storage devices (eg, data store 104) in computer system 101. Thus, it should be understood that computer storage devices may be included in computer system components that also (or even primarily) use transmission media.

[0034] Computer-executable instructions comprise, for example, instructions and data that, when executed in processing device(s) 102, cause computer system 101 to perform a specific function or set of functions. The computer-executable instructions may be, for example, binary data, instructions in an intermediate format such as assembly language, or even source code.

[0035] Those skilled in the art will appreciate that the present invention may be practiced in networked computing environments with many types of computer system configurations, including personal computers, desktop computers, laptop computers, messaging devices, portable devices, multiprocessor systems, microprocessor or programmable consumer electronics, network PCs, minicomputers, large multi-user computers, mobile phones, PDAs, tablets, paging devices, routers, switches, and the like. The present invention may also be practiced in distributed system environments where tasks are performed by both local and remote computer systems that are linked (via either wired data links, wireless data links, or a combination of wired and wireless data links) through the network. As such, in a distributed systems environment, a computer system may include a plurality of component computer systems. In a distributed systems environment, program modules may be located in both local and remote storage devices.

[0036] Those skilled in the art will also appreciate that the present invention may be practiced in a cloud computing environment. Cloud computing environments may be distributed, although this is not required. When distributed, cloud computing environments may be distributed throughout the world within an organization and/or have components owned by different organizations. In this specification and the following claims, "cloud computing" is defined as a model for providing on-demand network access to a shared pool of configurable computing resources (eg, networks, hosts, storage, applications, and services). The definition of "cloud computing" is not limited to any of the many other benefits that can be gained from such a model when properly deployed.

[0037] A cloud computing model may be composed of various characteristics such as on-demand self-service, free network access, pooling of resources, adaptability, measurable service, and so on. The cloud computing model can also take the form of different service models such as Software as a Service (SaaS), Platform as a Service (PaaS) and Infrastructure as a Service (IaaS) for example. The cloud computing model can also be deployed using different models deployments such as private cloud, community cloud, public cloud, hybrid cloud, and so on.

[0038] Some embodiments, such as a cloud computing environment, may comprise a system that includes one or more nodes, each capable of running one or more virtual machines. While running, virtual machines emulate a live computing system, supporting an operating system and possibly one or more other applications as well. In some embodiments, each node includes a hypervisor (operating system manager) that emulates virtual resources for virtual machines using physical resources that are abstracted to virtual machines. The hypervisor also provides proper isolation between virtual machines. Thus, from the point of view of any given virtual machine, the hypervisor gives the illusion that the virtual machine is interacting with a physical resource, even though the virtual machine is only interacting with an external representation (eg, virtual resource) of the physical resource. Examples of physical resources include processing power, memory, disk space, network bandwidth, media storage, and so on.

[0039] As shown, data store 104 may store computer-executable instructions and/or data structures representing application programs, such as, for example, tracer 104a, operating system kernel 104b, and application 104c (for example, an application that is a trace object for tracer 104a and one or more trace files 104d). When these programs are executed (eg, using processor 102), system memory 103 may store appropriate run-time data such as run-time data structures, computer-executable instructions, and so on. Therefore, Fig. 1 shows system memory 103 as including runtime program code 103a and application runtime data 103b (eg, which is associated with application 104c).

[0040] The tracer 104a may be used to record a bit-accurate trace of the execution of an application, such as application 104c, and store the trace data in the trace file(s) 104d. In some embodiments, the tracer 104a is a standalone application, while in other embodiments, the tracer 104a is integrated into another software component, such as an operating system kernel 104b, a hypervisor, a cloud, and so on. Although trace file(s) 104d are depicted as being stored in data store 104, trace file(s) 104d may also be written exclusively or temporarily in system memory 103 or some other storage device.

[0041] FIG. 1 includes a simplified representation of the internal hardware components of the processor(s) 102. As shown, each processor 102 includes a plurality of processing units 102a. Each processing unit can be physical (i.e., the physical core of a processing device) and/or logical (i.e., a logical core represented by a physical core that supports hyperparallelism, whereby more than one application execution thread runs on a physical core). Thus, for example, even though the processor 102 in some embodiments may include only one physical processing unit (core), it could include two or more logical processing units 102a represented by that one physical processing unit. .

[0042] Each processing unit 102a executes processor instructions that are specified by applications (e.g., tracer 104a, operating core 104b, application 104c, etc.), and such instructions are selected from a predefined instruction set architecture (ISA). architecture) of the processing device. The specific ISA of each processor 102 varies by processor manufacturer and processor model. Common ISAs include the IA-64 and IA-32 architectures from INTEL, INC., the AMD64 architecture from ADVANCED MICRO DEVICES, INC. and various Advanced RISC Machine ("ARM") architectures from ARM HOLDINGS, PLC, although a large number of other ISAs exist and can be used by the present invention. In general, an "instruction" is the smallest externally visible (ie, external to the processor) unit of code that can be executed by the processor.

[0043] Each processing unit 102a receives processor instructions from shared cache 102b and executes processor instructions based on data in shared cache 102b, based on data in registers 102d, and/or no input. In general, shared cache 102b is a small amount of main memory (i.e., small compared to the normal amount of system memory 103) that stores copies on the processor of portions of a backing store, such as system memory 103 and/or other cache. For example, when executing application code 103a, shared cache 102b holds portions of application runtime data 103b. If the processing unit(s) 102a require data not yet stored in shared cache 102b, then a "cache miss" occurs and the data is retrieved from system memory 103 (probably "pushing out" some other data from shared cache 102b).

[0044] Typically, shared cache 102b contains a plurality of "cache lines", each of which stores a piece of memory from a backing store. For example, Fig. 2 shows an example of at least part of a shared cache 200 that includes a plurality of cache lines 203 each including an address part 201 and a value part 202. The address portion 201 of each cache line 203 may store the address in the backing store (eg, system memory 103) that the line corresponds to, and the value portion 202 may initially store the value received from the backing store. The value portion 202 may be modified by the processing units 102a, and subsequently pushed back into the backing store. As indicated by the ellipses, shared cache 200 may include a large number of cache lines. For example, a modern INTEL processor may contain a level 1 cache containing 512 or more cache lines. In this cache, each cache line can typically be used to store a 64-byte (512-bit) value with a reference to an 8-byte (64-bit) memory address.

[0045] The address stored in the address portion 201 of each cache line 203 may be a physical address, such as an actual memory address in system memory 103. Alternatively, the address stored in the address portion 201 of each cache line 203 may be a virtual address , which is the address assigned to the physical address to provide abstraction. Such abstractions can be used, for example, to help isolate memory between different processes executing on processor(s) 102. When using virtual addresses, processor 102 can include a translation lookaside buffer (TLB) 102f (typically part of a memory management unit (MMU) that maintains a mapping between physical and virtual memory addresses.

[0046] Shared cache 102b may include a code cache part and a data cache part. For example, when executing application code 103a, the code portion of the shared cache 102b stores at least a portion of the processor instructions stored in the application code 103a, and the data portion of the shared cache 102b stores at least a portion of the data structures from the runtime data 103b. applications. Often the processor cache is split into separate tiers/levels (e.g., level 1 (L1), level 2 (L2), and level 3 (L3)), with some tiers (e.g., L3) theoretically existing separately from the processor (in ) 102. Thus, shared cache 102b may contain one of these levels (L1) or may contain multiple of these levels.

[0047] When using multiple cache levels, the processing unit(s) 102a interact directly with the lowest level (L1). In most cases, data flows between layers (for example, on a read, the L3 cache interacts with system memory 103 and feeds data to the L2 cache, and the L2 cache in turn feeds data to the L1 cache). When the processing unit 102a is about to write, the caches are coordinated to ensure that those caches that had changed data shared by the processing unit(s) 102a no longer have it. This coordination is done using the Cache Coherence Protocol (discussed later).

[0048] Caches can be inclusive, exclusive, or include both inclusive and exclusive behavior. For example, in an inclusive cache, the L3 layer would store a superset of the data in the L2 layers below it, and the L2 layers would store a superset of the L1 layers below it. In exclusive caches, levels can be non-overlapping, for example, if the L3 cache has data that the L1 cache needs, they can exchange information such as data, address, and the like.

[0049] Each processing unit 102 also includes firmware 102c, which contains control logic (i.e., executable instructions) that controls the operation of the processing device 102, and which typically acts as an interpreter between the processing device hardware and the processing device ISA, disclosed by the processing device 102 for executing applications. The firmware 102 may be practiced in a storage device embedded in the processing device such as ROM, EEPROM, etc.

[0050] Registers 102d are hardware storage locations that are defined based on the ISA of the processor(s) 102 and read from and/or written to by instructions from the processor. For example, registers 102d are typically used to store values retrieved from shared cache 102b for use by instructions, to store results of instruction execution, and/or to store status or state, such as some side effects of instruction execution (e.g., indication of value change, achievement value of zero, the occurrence of a carry, etc.), the processor cycle counter, etc. Therefore, some registers 102d may contain "flags" that are used to signal some state change caused by the execution of processor instructions. In some embodiments, processors 102 may also include control registers that are used to control various aspects of the processor's operation.

[0051] In some embodiments, processor(s) 102 may include one or more buffers 102e. As will be discussed herein below, buffer(s) 102e may be used as a temporary storage location for trace data. Thus, for example, processor(s) 102 may store portions of trace data in buffer(s) 102e, and flush that data to trace file(s) 104d at an appropriate time, such as when memory bus bandwidth is not busy. In some implementations, buffer(s) 102e may be part of a shared cache 102b.

[0052] As mentioned above, processors having a shared cache 102b operate the cache according to the Cache Coherence Protocol (CCP). Specifically, the CCP specifies how consistency is maintained between data in shared cache 102b and a backing data store (e.g., system memory 103 or other cache) as different processing units 102a read and write data in shared cache 102b, and how it is ensured that different processing units 102a have always read valid data from a given location in shared cache 102b. Typically, CCPs are associated with and adapted to the memory model specified by the processor 102 ISA.

[0053] Examples of common CCPs include the MSI protocol (i.e., Modified, Shared, and Invalid), the MESI protocol (i.e., Modified, Exclusive, Shared, and Invalid - Modified, Exclusive, Shared and Invalid), and the MOESI protocol (i.e. Modified, Owned, Exclusive, Shared, and Invalid). Each of these protocols defines the state for individual cells (eg, rows) in shared cache 102b. A "modified" cache location contains data that has been modified in shared cache 102b and is therefore likely not consistent with corresponding data in a backing store (eg, system memory 103 or another cache). When a cell in the "modified" state is evicted from shared cache 102b, the common CCPs require the cache to ensure that its data is written back to the backing store, or that another cache takes over the responsibility. A "shared" cache location contains data that is not modified from backing data, is in a read-only state, and is shared by processing unit(s) 102a. The shared cache 102b can evict this data without writing it to the backing store. An "invalid" cache location does not contain valid data and can be considered empty and usable for storing data as a result of a cache miss. The "exclusive" cache location contains data that matches the backing store and is used by only one processing unit 102a. It may be changed to the "shared" state at any time (ie, in response to a read request), or may be changed to the "modified" state when written to it. The "own" cache location is shared by two or more processors 102a, but one of the processors has the exclusive right to make changes to it. When such a processing unit makes changes, it directly or indirectly notifies other processing units, as notifiable processing units may need to be invalidated or updated, depending on the CCP implementation.

[0054] The granularity with which different CCPs monitor cache coherency and make this cache coherency data available to tracer 104a may vary. For example, at one end of the range, some CCPs keep track of cache coherence for each cache line as well as for each processing unit. Therefore, these CCPs can keep track of the state of each cache line, and how it is related to each processing unit. As will be shown in the example that follows with respect to FIG. 6A-6D, this means that an individual cache line may have information about its state and how it is associated with each processing unit 102a. Other CCPs are less granular and only track cache coherence at the cache line level (and don't have per-processor information). At the other end of the range, processor vendors may choose to track cache coherence only at the cache line level for efficiency, since only one processor at a time can exclusively own a line (exclusive, modified, etc.). As an example of medium granularity, the CCP can keep track of the cache coherency for each cache line, as well as an index (eg, 0, 1, 2, 3 for a processor with four processing units) for the processing unit that has the current state of the cache line.

[0055] Embodiments employ the processor's shared cache 102b to efficiently capture bit-accurate execution trace of application 104c and/or operating system kernel 104b. These embodiments build on the observation that the processor 102 (including the shared cache 102b) forms a semi- or semi-closed system. For example, after loading a portion of the data to be processed (i.e., code data and runtime application data) into the shared cache 102b, the processor 102 may operate on its own, without any input, as a semi- or quasi-closed system for short time intervals. In particular, one or more processing units 102a execute instructions from the code portion of the shared cache 102b using the runtime data stored in the data portions of the shared cache 102b and using the registers 102d.

[0056] When processing unit 102a needs some information to arrive (for example, because an instruction it executes, will execute, or can execute accesses code data or run-time data that is not yet in shared cache 102b), a "cache -miss", and this information is pushed into shared cache 102b from system memory 103. For example, if a data cache miss occurs when an executable instruction performs an operation in memory at a memory address within application runtime data 103b, data from that memory address is pushed to one of the cache lines of the data portion of the shared cache 102b. Similarly, if a code cache miss occurs when an instruction performs an operation in memory at application code 103a with a memory address stored in system memory 103, the code from that memory address is populated into one of the cache lines of the code portion of the shared cache 102b. Processing unit 102a then continues execution using the new information in shared cache 102b until the new information is re-populated into shared cache 102b (eg, due to another cache miss or an uncached read).

[0057] The present inventor has observed that in order to record a bit-accurate representation of the execution of the application, the tracer 104a can record enough data to be able to repeat the arrival of the information in the shared cache 102b during the execution of that application execution thread(s). The first approach to this is to record all data that is entered into the shared cache 102b by logging all cache misses and non-cached reads (i.e., reads from hardware components and non-cached memory), as well as time during execution, when each piece of data has been entered into the shared cache 102b (eg, using the number of instructions executed or some other counter).

[0058] A second approach, which results in significantly smaller trace files than the first approach, is to track and record the cache lines that were "consumed" by each processing unit 102a. As used herein, a processing unit has "consumed" a cache line if it knows its current value. This may be because the processing unit is the one that wrote the current value of the cache line, or because the processing unit performed a read on the cache line. This second approach encompasses shared cache extensions 102b that allow processor 102 to identify, for each cache line, one or more processors 102a that have consumed the cache line.

[0059] In accordance with embodiments herein, a third approach is to use the processor's CCP to determine a subset of "consumed" cache lines to write to file(s) 104d, and this will still allow shared cache actions to be repeated. 102b. This third approach results in much smaller trace files, and therefore much less overhead, than both the first and second approaches.

[0060] Some embodiments herein record trace data streams that correspond to processing units/flows of execution. For example, trace file(s) 104 may include one or more separate trace data streams for each processing unit. In these embodiments, the data packets in each trace data stream may not have an identification of the processing unit to which the data packet belongs, since this information is inherent based on the trace data stream itself. In these embodiments, if the computer system 101 includes multiple processors 102 (i.e., in different processor slots), the trace file(s) may have one or more different trace data streams for each processing unit 102a in different processing devices 102. Multiple data streams can be used even for a single thread of execution. For example, some embodiments may associate one data stream with a processing unit used by a thread of execution, and associate one or more additional data streams with each shared cache used by a thread of execution.

[0061] In other embodiments, trace file(s) 104 may include one trace data stream for processor 102 and may identify in each data packet which processing unit the data packet belongs to. In these embodiments, if the computer system 101 includes multiple processors 102, the trace file(s) 104 may include a separate trace data stream for each of the multiple processors 102. Regardless of the format of the trace file(s), the data packets for each processing unit 102a are generally recorded independently of other processing units, which allows different threads of execution that were executed in different processing units 102a to be independently reproduced. However, trace files may include some information, whether written or native, that provides partial ordering between different threads of execution.

[0062] FIG. 3 shows a flow diagram of a method 300 for performing a cache-based trace entry using CCP data. Method 300 may include steps that are performed by processor 102 as tracer 104a traces application 104c and/or operating system kernel 104b. The actions performed by processor 102 may be based on embedded logic in processor 102, software coded logic (i.e., firmware 102c), and/or other software application such as tracer 104a, operating system kernel 104b, or hypervisor. Although Fig. 3 and shows the sequence of steps, it should be understood that embodiments may perform many of these steps in any order, with some even being performed in parallel. Accordingly, the sequence of steps illustrated in method 300 is not limiting.

[0063] As depicted, method 300 includes step 301, which detects an interaction between the cache and the backing store. In some embodiments, step 301 includes detecting an operation that causes an interaction between a particular cache line from a plurality of cache lines and one or more backing stores. For example, when executing an application thread 104c or an operating system kernel 104b in one of the processing units 102a, the processing unit may cause an interaction between a line in shared cache 102b and a backing store (eg, system memory 103 or another cache). The detection may be performed, for example, by the processor 102 based on the execution of its firmware 102c.

[0064] Method 300 also includes step 302, which identifies the processing unit that caused the interaction. In some embodiments, step 302 includes identifying a particular processing unit from a plurality of processing units that caused the operation. For example, based on the execution of the firmware 102c, the processor 102 may identify which of the processing units 102a caused the operation detected in step 301.

[0065] The method 300 also includes step 303 determining whether registration is enabled for the processing unit. In some embodiments, step 303 comprises using one or more registration control bits to determine that registration is enabled for a particular processing unit. For example, processor 102 may determine whether the processing unit identified at block 302 has registration enabled based on one or more registration control bits. The use of registration control bit(s) allows you to dynamically enable and disable registration of different processing units. Thus, through the use of the registration control bit(s), tracer 104a can dynamically control which thread(s) of execution to trace and/or which portion(s) of execution of different threads of execution to trace.

[0066] The specific form and function of the registration control bit(s) may vary. In some embodiments, for example, the registration control bit(s) is/are an integral part of one of the registers 102d, such as a control register. In these embodiments, one registration control bit may correspond to one processing unit 102a or multiple processing units 102a. Accordingly, register 102d includes a single registration control bit (e.g., corresponding to all processing units, or a particular processing unit, or a subset of processing units), or may potentially include multiple registration control bits (e.g., each corresponding to one or more processing blocks). In other embodiments, the registration control bit(s) contains, or is otherwise associated with, an address space identifier (ASID) and/or a process-context identifier (PCID) corresponding to the instruction that caused the interaction. between cache and backing store. Accordingly, for example, method 300 may only trace a processing block when it executes instructions associated with one or more particular ASID/PCID. Thus, method 300 may only record specific address space(s) and/or specific process context(s). Combinations are also possible. For example, the registration control bit(s) may be stored in one or more registers 102d and set/cleared based on the current ASID/PCID values. Regardless of the form of the registration control bit(s), some embodiments may be able to set/clear the registration control bit(s) on context switches, allowing method 300 to trace only specific threads of execution.

[0067] Method 300 also includes step 304 determining whether a cache line participates in registration. In some embodiments, step 304 comprises, based at least on the fact that registration is allowed for a particular processing unit, determining whether a particular cache line participates in registration. For example, processor 102 can determine if a cache line involved in the operation detected in step 301 is registered. As will be discussed in more detail below, there are several mechanisms that can be used for detection, such as the use of bits within shared cache 102b. or using cache login blocking.

[0068] The method 300 also includes step 305, which uses the CCP to identify that there is data to be logged in the trace. For example, processor 102 may refer to its CCP to determine which cache state transitions occurred as a result of an operation and whether those transitions are the basis for data logging. Detailed examples of using CCP to identify trace data are provided below with respect to FIG. 6A-9B.

[0069] The method 300 also includes step 306, which logs the appropriate data in the trace using the CCP. In some embodiments, step 306 includes causing data to be logged in the trace, which data can be used to reproduce the operation. When data is to be logged in the trace file(s), one or more data packets can be added to the appropriate trace data streams, such as the trace data stream that corresponds to a particular processing unit, or the trace data stream that corresponds to the processing device 102 as a whole. . If the proper trace data stream matches the processor 102 as a whole, one or more data packets may identify a particular processing unit. It should be noted that if the trace data stream corresponds to the processor 102 as a whole, the inherent order of the data packets in the data stream itself provides some additional ordering information that may not be available if multiple data streams are used.

[0070] It should be noted that when shared cache 102b contains multiple cache levels, in some embodiments, method 300 operates at a cache level that interacts with system memory 103, since it is this cache level that handles cache misses. Operating at this level allows the cache actions of each processing block 102a to be represented without redundancy (ie, not presenting the block actions more than once). Thus, for example, if computer system 101 includes two processors 102 (i.e., two processor slots) and contains one "inclusive" L3 cache for each slot, as well as "inclusive" L2 caches below the L3 cache , in some embodiments, method 300 operates on L3 caches. Method 300 may also operate on multiple cache levels. For example, if computer system 101 includes one processor 102 (i.e., one socket for installing processors) and contains one "exclusive" L3 cache per socket, as well as "inclusive" L2 caches below the L3 cache, method 300 may work on both L3 and L2 caches. Additional examples of registering within caches showing mixed inclusive/exclusive behavior are discussed below.

[0071] As mentioned above with respect to block 304, there are several mechanisms that can be used by processor 102 to determine if a cache line is a "register member". One is to expand each line of the shared cache 102b with one or more additional "account bits" that can be used as a flag, as processor identifiers, or as a processor index. The logic to control these "account bits" may be part of the processor firmware 102c.

[0072] To show this embodiment, FIG. 4A shows an exemplary shared cache 400a similar to the shared cache 200 shown in FIG. 2 that extends each of its cache lines 404 with one or more additional account bits 401. Thus, each cache line 404 includes account bit(s) 401, regular address bits 402, and value bits 403.

[0073] In some embodiments, the accounting bit(s) 401 of each cache line contains one bit that functions as a flag (i.e., on or off) used by processor 102 to indicate whether or not the cache line participates in trace logging. If the processor's CCP is granular enough (for example, if the CCP keeps track of the coherence state for each cache line, either as its relation to each processing unit, or by referring to an index for the processing unit to which the cache line's coherence state belongs), this single bit may be sufficient to ensure that a reliable, fully deterministic trace is recorded (i.e., one that guarantees the ability to completely reconstruct the traced execution).

[0074] In other implementations, the accounting bit(s) 401 of each row includes a plurality of bits. The set of bits can be used in several ways. Using one approach, referred to herein as "block bits", the accounting bit(s) 401 of each cache line may include a number of block bits equal to the number of processing units 102a of the processor 102 (e.g., the number of logical processing units if the processing device 102 supports hyperparallelism, or the number of physical processing units if hyperparallelism is not supported). These block bits can be used by the processor 102 to keep track of which one or more particular processing units have consumed the cache line (or, if the cache line has not been consumed, to indicate that none of the processing units have consumed it). Thus, for example, a shared cache 102b that is shared by two processing units 102a may include two block bits for each cache line. In connection with these block bits added to each cache line, embodiments extend processor firmware 102c to use these block bits to keep track of whether the current value in the cache line has been registered (i.e., in trace file 104d) on behalf of each processing unit, or otherwise known to the processing unit. If the processor's CCP has a coarser granularity (for example, if the CCP only monitors the coherency state at the cache line level), these block bits can provide additional information to provide reliable tracing. For example, if a cache line is marked as shared or exclusive by the CCP, the block bits can be used to identify which processing unit(s) shares the cache line, or which processing unit has exclusive rights.

[0075] Using another approach, referred to herein as "index bits", the accounting bit(s) 401 of each cache line may include a number of index bits sufficient to represent an index for each of the processor processing units 102a (in ) 102 of the computer system 101 that participates in the registration, along with a "reserved" value (eg, -1). For example, if the processor(s) 102 of computer system 101 includes 128 processing units 102a, these processing units can be identified by an index value (eg, 0-127) using only seven index bits per cache line. In some embodiments, one index value is reserved (eg, "invalid") to indicate that no processor has registered a cache line. Accordingly, this would mean that seven index bits could actually represent 127 processing units 102a plus the reserved value. For example, the binary values 0000000-1111110 could correspond to index locations 0-126 (in decimal), and the binary value 1111111 (for example, -1 or 127 in decimal, depending on interpretation) could correspond to "invalid" to indicate, that no processor has registered the corresponding cache line, although this designation may vary, depending on the implementation. Thus, the block bits may be used by processor 102 to keep track of whether a cache line is participating in a trace log (eg, a value other than -1) and as an index to the particular processing block that consumed the cache line (eg, a processing block that last consumed it). This second approach has the advantage of supporting a large number of processing units at low overhead in shared cache 102b, with the disadvantage of being less granular than the first approach (i.e., only one processing unit is identified at a time). Again, if the processor's CCP has a coarser granularity (for example, if the CCP only monitors the coherence state at the cache line level), these index bits can provide additional information to provide reliable tracing. For example, if a cache line is marked as shared or exclusive by the CCP, the index bits can be used to identify at least one processing unit that shares the cache line, or which processing unit has exclusive rights.

[0076] Another mechanism that may be used by processor 102 to determine whether a cache line is a registration member may use the concepts discussed in connection with FIG. 4A, but without expanding each cache line with additional accounting bit(s) 401 for accounting bits. Instead, this mechanism reserves one or more cache lines 404 to store accounting bits. Fig. 4B shows an example of a shared cache 400b that includes regular cache lines 405 that store memory addresses 402 and values 403, as well as one or more reserved cache lines 406 for storing accounting bits that apply to regular cache lines 405. The bits of the reserved cache line(s) 406 are allocated to different groups of accounting bits, each of which corresponds to one distinct of the regular cache lines 405 . These groups of accounting bits may function as a flag bit, block bits, or index bits, depending on the implementation.

[0077] Another mechanism that can be used by the processor(s) 102 to determine if a cache line is a registration member is to use associative caches and login blocking. Because processor shared cache 102b is typically much smaller than system memory 103 (often by orders of magnitude), there are typically many more memory locations in system memory 103 than there are lines in shared cache 102b. Therefore, each processor defines a mechanism for mapping multiple system memory locations to line(s) in a shared cache. Generally, processors use one of two general methods: direct mapping and associative mapping. When direct mapping is used, different memory locations in system memory 103 are mapped to only one cache line, so that each memory location can only be cached to a specific cache line.

[0078] On the other hand, when using associative mapping, different cells in the system memory 103 can be cached into one of the many lines in the shared cache 102b. Fig. 5 shows an example 500 associative cache mappings. In this case, cache lines 504 from cache 502 are logically divided into different address groups of two cache lines each, including a first group of two cache lines 504a and 504b (identified at index 0) and a second address group of two cache lines 504c and 504d. cache (identified with index 1). Each cache line in the address group is associated with its "entry", such that cache line 504a is identified by index 0, entry 0, cache line 504b is identified by index 0, entry 1, and so on. As shown further, memory locations 503a, 503c, 503e, and 503g (memory indexes 0, 2, 4, and 6) are mapped to index 0. Accordingly, each of these locations in system memory can be cached to any cache line within the group at index 0. (ie, cache lines 504a and 504b). The particular mapping patterns depicted are for illustrative and conceptual purposes only and should not be interpreted as the only way memory indexes can be mapped to cache lines.

[0079] Associative caches are generally referred to as N-entry associative caches, where N is the number of "entries" in each address group. Thus, cache 500 in FIG. 5 may be referred to as a 2-entry associative cache. Processors typically implement N-entry caches, where N is a power of two (e.g., 2, 4, 8, etc.), with N typically chosen from 4 and 8 (although the embodiments herein are not limited to any particular N values or subsets of N values). In particular, a 1-entry associative cache is generally equivalent to a direct-mapped cache, since each address group contains only one cache line. Additionally, if N is equal to the number of lines in the cache, it is referred to as a fully associative cache because it contains a single address group containing all the lines in the cache. In fully associative caches, any memory location can be cached to any line in the cache.

[0080] It should be noted that FIG. 5 is a simplified view of system memory and caches to illustrate the general principles. For example, although in FIG. 5 individual memory locations are mapped to cache lines, it should be understood that each cache line typically stores data related to multiple addressable locations in system memory. Thus, in FIG. 5, each cell (503a-503h) in the system memory (501) may actually represent a plurality of addressable memory cells. Additionally, the mappings may be between actual physical addresses in system memory 501 and lines in cache 502, or may use an intermediate level of virtual addresses.

[0081] Associative caches can be used to determine if a cache line is a member of the registration, through the use of login blocking. Entry blocking blocks or reserves certain entries in the cache for some purpose. In particular, embodiments herein use entry blocking to reserve one or more entries for a thread of execution that is being traced, such that blocked/reserved entries are used solely to store cache misses related to the execution of that thread of execution. Thus, returning to FIG. 5, if "input 0" was blocked for a traced thread of execution, then cache lines 504a and 504c (i.e., index 0, input 0 and index 1, input 0) will be used exclusively for cache misses related to the execution of that thread. execution, and the rest of the cache lines will be used for all other cache misses. Thus, to determine if a particular cache line is a registration member, the processor 102 need only determine if the cache line is part of an input that is reserved for the thread of execution that is being traced.

[0082] FIG. 6A-6D show a specific example 600 of applying the method 300 shown in FIG. 3 in the context of FIG. 1, 2, 4A, 4B and 5. FIG. 6A shows a first table 600a that shows the read and write operations of four processing units 102a (ie, P0-P3) on a single line in a shared cache 102b. Fig. 6B shows a second table 600b that indicates one embodiment of the monitored cache coherence state (eg, as monitored using a CCP processor) based on these reads and writes. Fig. 6C shows a third table 600c that shows what could be stored in the account bits of shared cache 102b (as described with reference to FIGS. 4A and 4B), if account bits are used at all. Although only one type of accounting bit is typically used (i.e., block bits for each row, index bits for each row, or flag bit for each row), for completeness, table 600c shows all, block bits 603, index bits 604, and flag bit 605. Finally, FIG. 6D shows a fourth table 600d that shows exemplary log data types 606 that can potentially be written to trace file(s) 104d with respect to each operation.

[0083] For ease of description, table 600a depicts operations of only one processing unit 102a at a time, but it should be understood that the principles described herein apply to situations where there are simultaneous actions (e.g., simultaneous readings by two or more processing units of one and the same cache line). Additionally, the examples described with respect to FIG. 6A-6D assume that tracking is enabled for processing units P0-P2 and disabled for processing unit P3. For example, as discussed above, this may be controlled by a bit corresponding to each processing block, such as a plurality of control register bits.

[0084] Initially, for ease of description, this example will use the simplified cache line states that are derived from the cache line states (i.e., Modified, Own, Exclusive, Shared, and Invalid) used in the CCPs discussed above (i.e., e. MSI, MESI and MOESI). With this simplification, these states map to either a "read" state (ie, the cache line has been read) or a "write" state (ie, the cache line has been written). Table 1 below shows one example of these mappings. It should be noted that these mappings are used by way of example only and are not limiting. For example, there may be CCPs and states other than those discussed herein, and those skilled in the art will appreciate, given the disclosure herein, that similar mappings can be made for many different CCPs.

State in protocol Displayed state Modified Recording Own Reading Exclusive Recording shared Reading Invalid No display, cache line is considered empty

Table 1

[0085] Significantly, embodiments may log CCP data at various levels, depending on what data is available from processor 102, and/or based on implementation choices. For example, CCP data may be logged based on "mapped" CCP states (such as those shown in Table 1), based on actual CCP states (Modified, Proprietary, Exclusive, Shared, and/or Invalid) that are visible to processor 102, and/ or even based on lower level "raw" CCP data that cannot normally be seen by processor 102.

[0086] Referring to FIG. 6A-6D, table 600a includes a first column 601 showing an identifier (ID) that is used to determine a global order among operations. Table 600a also includes four additional columns 602a-602d, each corresponding to one of the processing units. Although this example uses a global ID for simplicity, it should be understood that in practice each processing unit typically marshals operations using its own independent set of identifiers. These IDs may contain an instruction count (IC) or any other suitable mechanism for determining the order among operations, such as a "hop counter" plus a program counter. Note that in the example the memory is used in such a way as to be consistent with the MSI, MESI and MOESI CPPs, but for simplicity only the "modified", "shared" and "invalid" states are used. However, it should be noted that some CCPs may provide their own unique and/or incremental IDs, which may also be recorded in the trace (eg, in each packet, or in random packets) in order to strictly order trace elements. Even if the CCP does not provide such an ID, a socket timer value (such as TSC) or another orderable ID could potentially be used.

[0087] As shown in Table 600a, at ID[0], the processing unit P0 performs a read that causes a cache miss, resulting in data DATA[1] being entered into the cache line. Accordingly, table 600b shows that the processor CCP notes that the cache line is now "shared" by the P0 block. Table 600c shows that if block bits 603 are used, they indicate that processing block P0 has consumed (i.e., read) a cache line (and that processing blocks P1-P3 have not consumed), that if index bits 604 are used, they indicate, that P0 has consumed a cache line, and that if flag bit 605 is used, it indicates that some processing unit has consumed a cache line. Given this status, at step 303, the processor 102 determines that the registration is allowed for P0, and at step 304 it determines that the cache line participates in the registration (i.e., using block bits 603, index bits 604, flag bit 605, or lock entrance). Accordingly, at step 306, the processor 102 invokes the CCP to log the appropriate data to the trace file(s), if necessary. In this case, since the cache line transitions from the invalid (empty) state to the read (table 600a)/shared (table 600b) state, data must be registered. As shown in logging data 606 in table 600d, processor 102 may flag a processing unit (P0) as needed (i.e., depending on whether data packets are registered for separate data streams per processing unit, or for a single data stream ); address (@) cache line; instruction counter or some other counter; and the data (DATA[1]) that was entered into the cache line. Although, as discussed above, the instruction counter will typically be a processing unit-specific value, for simplicity, table 600d refers to the instruction counter by referring to the corresponding global ID (ie, IC[0], in this case).

[0088] It should be noted that the cache line address (@) of the cache line and data (eg, DATA[1]) in some embodiments may be compressed in trace file(s) 104d. For example, memory addresses can be compressed by not writing the "high" bits of the memory address, by binding (explicitly or implicitly) to the "high" bits in a previously written memory address. Data can be compressed by grouping the data value bits into a plurality of groups, each containing a plurality of bits, and associating each group with a corresponding "flag" bit. If the group equals a particular pattern (eg, all 0s, all 1s, etc.), a flag bit may be set, and this group of bits shall not be stored in the trace.

[0089] Next, table 600a shows that, at ID[1], the processing unit P1 performs a read in the cache line, reading data DATA[1]. Table 600b shows that the processor CCP notes that the cache line is now "shared" by blocks P0 and P1. Table 600c shows that these processing units P0 and P1 consumed a cache line (block bits 603), that P1 consumed a cache line (index bits 604), or that some processing unit consumed a cache line (flag bit 605). Note that for index bits 604 it would also be correct to still refer to P0 instead of P1. Table 600d shows that, using the CCP, the processor 102 determines that a transaction record should be registered. As shown, the processor 102 may flag a processing unit (P1); address (@) cache line; instruction counter (IC[1]); that the cache line has moved from the read (shared) state to the read (shared) state; and that P0 had previous access to the previous cache line, but now P0 and P1 have access.

[0090] Next, table 600a shows that, at ID[2], the processing unit P0 writes to the cache line, writing data DATA[2]. Table 600b shows that the processor CCP notes that the cache line is now "modified" by block P0 and "invalid" for P1. Table 600c shows that only processing unit P0 has consumed (i.e., updated its value) a cache line (block bits 603), that P0 has consumed a cache line (index bits 604), or that some processing unit has consumed a cache line (flag bit 605). Table 600d shows that, using the CCP, the processor 102 determines that an operation entry should be registered since the cache line has been written/modified. As shown, processor 102 may flag a processing block (P0); address (@) cache line; instruction counter (IC[2]); that the cache line has moved from a read (shared) state to a write (modified) state; and that P0 and P1 had previous access to the previous cache line, but now only P0 has access.

[0091] It should be noted that information about which processing unit(s) had previous access to the cache line could be found using the state in the CCP shown in table 600b. However, it should be noted that some CCPs may not maintain enough information to do this (for example, a CCP that only keeps track of the coherence state at the cache line level). Alternatively, if block bits 603 are used, this information can be derived from the block bits. Accordingly, the registration data 606 shown in FIG. 6D assume either a reliable CCP that supports this information, or the use of block bits 603.

[0092] If none of these are used (for example, if the CCP is not as reliable, and if index bits 604, flag bit 605, or entry blocking are used instead of block bits 603), registration data 606 may be less complete or larger. As a first example, if the CCP only monitors the coherence state at the cache line level, and if index bits 604 are used, they can be used to identify that the cache line state is invalid (for all processing units), that it is modified (together with the index of the processing unit that modified it), that it is exclusive (together with the index of the processing unit that has exclusive rights), or that it is shared (and all processing units can access). This can lead to a simpler hardware implementation, with the disadvantage that when a cache line needs to be changed from shared to modified or exclusive, all processing units must be notified, not just those that would be known to the CCP in greater detail as sharing the line. cache. As a second example, index bits 604 may be used to identify the last processing unit that accessed the cache line. Then, if the cache is inclusive (i.e., many reads are hidden behind accesses at the L2 or L1 cache levels), then even if the processing units read the same cache line, the L3 cache may see relatively few repeated requests from the same processing blocks. When logging each index change for read->read and then logging read->write, write->write and write->read, the index just as well yields the same data as using block bits 603, due to tracing, probably a little bigger size. As a third example, each cache line may include one flag bit, but the CCP may keep track of the coherence state for each cache line against an index for the processing unit that owns the cache line's coherence state. In this case, the trace may record more cache line state offsets than if block bits were used, or CCP traced individual processing blocks, but the trace could still be completely deterministic. A brief comparison of the size of the trace file with information about each processing unit versus only information about the processor index is provided later in this document with reference to FIG. 9A and 9B.

[0093] Returning to FIG. 6A, table 600a shows that, at ID[3], the processing unit P1 reads from the cache line by reading data DATA[2]. Table 600b shows that the processor CCP notes that the cache line is now "shared" by blocks P0 and P1. Table 600c shows that processing units P0 and P1 consumed a cache line (block bits 603), that P1 consumed a cache line (index bits 604), or that some processing unit consumed a cache line (flag bit 605). Note that for index bits 604 it would also be correct to still refer to P0 instead of P1. Table 600d shows that, using the CCP, processor 102 determines that an operation record should be registered because the cache line has moved from a write (modified) state to a read (shared) state. As shown, the processor 102 may flag a processing unit (P1); address (@) cache line; instruction counter (IC[3]); that the cache line has moved from the write (modified) state to the read (shared) state; and that P0 had previous access to the previous cache line, but now P0 and P1 have access.

[0094] Next, table 600a shows that, at ID[4], the processing unit P0 writes to the cache line again, this time writing data DATA[3]. Table 600b shows that the processor CCP notes that the cache line is "modified" again by block P0 and "invalid" for P1. Table 600c shows that only processing unit P0 consumed a cache line (block bits 603), that P0 consumed a cache line (index bits 604), or that some processing unit consumed a cache line (flag bit 605). Table 600d shows that, using the CCP, the processor 102 determines that an operation record should be registered since the cache line has been written/modified. As shown, processor 102 may flag a processing block (P0); address (@) cache line; instruction counter (IC[4]); that the cache line has moved from a read (shared) state to a write (modified) state; and that P0 and P1 had previous access to the previous cache line, but now only P0 has access.

[0095] Next, table 600a shows that, at ID[5], the processing unit P2 performs a read from the cache line, reading data DATA[3]. Table 600b shows that the processor CCP notes that the cache line is now "shared" by blocks P0 and P2. Table 600c shows that processing units P0 and P2 consumed a cache line (block bits 603), that P2 consumed a cache line (index bits 604), or that some processing unit consumed a cache line (flag bit 605). Note that for index bits 604 it would also be correct to still refer to P0 instead of P2. Table 600d shows that, using the CCP, processor 102 determines that an operation record should be registered because the cache line has moved from a write (modified) state to a read (shared) state. As shown, processor 102 may flag a processing unit (P2); address (@) cache line; instruction counter (IC[5]); that the cache line has moved from the write (modified) state to the read (shared) state; and that P0 had previous access to the previous cache line, but now P0 and P2 have access.

[0096] Next, table 600a shows that, at ID[6], the processing unit P1 performs a read from the cache line, also reading data DATA[3]. Table 600b shows that the processor CCP notes that the cache line is now "shared" by blocks P0, P1, and P2. Table 600c shows that processing units P0, P1, and P2 consumed a cache line (block bits 603), that P1 consumed a cache line (index bits 604), or that some processing unit consumed a cache line (flag bit 605). Note that for index bits 604 it would also be correct to still refer to P0 or P2 instead of P1. Table 600d shows that, using the CCP, the processor 102 determines that a transaction record should be registered. As shown, the processor 102 may flag a processing unit (P1); address (@) cache line; instruction counter (IC[6]); that the cache line has moved from the read (shared) state to the read (shared) state; and that P0 and P2 had previous access to the previous cache line, but now P0, P1 and P2 have access.

[0097] Next, table 600a shows that, at ID[7], the processing unit P3 performs a read from the cache line, also reading data DATA[3]. Table 600b shows that the processor CCP notes that the cache line is now "shared" by blocks P0, P1, P2, and P3. Table 600c shows that none of the block bits 603, index bits 604, or flag bit 605 have been updated. This is because registration is disabled for P3 and, for tracing purposes, it thus did not "consume" a cache line when performing a read. Table 600d shows that no data has been logged. This is because, at step 303, the processor 102 will determine that registration is not allowed for P3.

[0098] Next, table 600a shows that, at ID[8], the processing unit P3 writes to the cache line, writing data DATA[4]. Table 600b shows that the processor CCP notes that the cache line is now "invalid" for P0, P1, and P2, and has been "modified" by the P3 block. Table 600c shows that block bits 603, index bits 604, and flag bit 605 all reflect the cache line as not being consumed by any processing block. This is because logging is disabled for P3, so, for tracing purposes, it does not "consume" a cache line when doing a write; in addition, the write invalidates the value in the cache line for other processing units. Table 600d shows that no data has been logged. Again, this is because, at step 303, the processor 102 will determine that the registration is not allowed for P3.

[0099] Next, table 600a shows that, at ID[9], the processing unit P0 writes to the cache line, writing data DATA[5]. Table 600b shows that the processor CCP notes that the cache line is now "modified" by block P0 and "invalid" for P3. Table 600c shows that no processing unit has consumed a cache line. This is because no registration element was entered in connection with this operation, as reflected in table 600d. No logging element should be entered because the written data will be repeated through the normal execution of the P0 block execution flow instructions. However, in some cases an item may be written to the trace under these circumstances (ie, a write to a cache line that is not registered by the processing unit when registration is enabled) to provide additional data to the trace consumer. Under these circumstances, the registration element can be thought of as a cache line value read plus a DATA[5] entry.

[00100] Next, table 600a shows that, at ID[10], the processing unit P2 performs a read from the cache line, reading data DATA[5]. Table 600b shows that the processor CCP notes that the cache line is now "shared" by blocks P0 and P2. Table 600c shows that processing unit P2 consumed a cache line (block bits 603), that P2 consumed a cache line (index bits 604), or that some processing unit consumed a cache line (flag bit 605). Table 600d shows that, using the CCP, processor 102 determines that an operation record should be registered because the cache line value was not previously registered (ie, it was not registered by ID[9]). As shown, processor 102 may flag a processing unit (P2); address (@) cache line; instruction counter (IC[10]); data (DATA[5]) that was entered into the cache line; and that P2 has access to the cache line. It may also be registered that P0 also has access to the cache line, depending on what information the particular CCP and accounting bits provide.

[00101] Next, table 600a shows that, at ID[11], the processing unit P1 performs a read from the cache line, also reading data DATA[5]. Table 600b shows that the processor CCP notes that the cache line is now "shared" by blocks P0, P1, and P2. Table 600c shows that processing units P1 and P2 consumed a cache line (block bits 603), that P1 consumed a cache line (index bits 604), or that some processing unit consumed a cache line (flag bit 605). Note that for index bits 604 it would also be correct to still refer to P2 instead of P1. Table 600d shows that, using the CCP, the processor 102 determines that a transaction record should be registered. As shown, the processor 102 may flag a processing unit (P1); address (@) cache line; instruction counter (IC[11]); that the cache line has moved from the read (shared) state to the read (shared) state; and that P2 had previous access to the previous cache line, but now P1 and P2 have access. Note that the value (DATA[5]) does not need to be registered since it was registered by the P2 block at ID[10].

[00102] Next, table 600a shows that, at ID[11], the processing unit P0 performs a read from the cache line, also reading data DATA[5]. Table 600b shows that the processor CCP notes that the cache line is now "shared" by blocks P0, P1, and P2. Table 600c shows that processing units P0, P1, and P2 consumed a cache line (block bits 603), that P0 consumed a cache line (index bits 604), or that some processing unit consumed a cache line (flag bit 605). Note that for index bits 604 it would also be correct to still refer to P1 or P2 instead of P0. Table 600d shows that, using the CCP, the processor 102 determines that a transaction record should be registered. In this case, the processing device 102 may mark the block (P0) processing; address (@) cache line; instruction counter (IC[12]); that the cache line has moved from the read (shared) state to the read (shared) state; and that P1 and P2 had previous access to the previous cache line, but now P0, P1 and P2 have access. The value (DATA[5]) is not logged because it is available from P2.

[00103] Alternatively, it is possible that the processor 102 only refers to P0 by ID[12] because P0 already has a cache line value (i.e., because it has stored that value by ID[9]). It is even possible to opt out of any registration on ID[12] since it is possible to use replay heuristics to recover the value (ie DATA[5]) without being aware that P0 is involved in the trace. However, these techniques can be computationally expensive and reduce the system's ability to detect when playback is "ripped". An example of a heuristic is the recognition that memory accesses of different processing units are generally strictly ordered (based on CCP data), so the last value in those units for a given memory location can be used during playback.

[00104] Next, table 600a shows that at ID[13] the cache line is being evicted. As a result, table 600b shows that the CCP entries are empty, table 600c shows that the accounting bits do not reflect processing units as having consumed a cache line, and table 600d shows that no data is logged.

[00105] It should be noted that while, for completeness, log data 606 lists all final access states (i.e., which processing units currently have access to a cache line), this information is potentially implicit and the size of the trace file can be reduced by discarding it. For example, in a write->read transition, the list of processing units that have access after the read always consists of the processing unit that had the previous access, plus the processing unit that performed the read. On a read->write transition or a write->write transition, the list of processing units that have write access after a write always consists of the processing unit that performed the write. In a read->read transition, the list of processing units that have access after the read always consists of the processing units that had access before the jump plus the processing unit that performed the read.

[00106] In general, in order to generate a fully deterministic trace file, CCP will dictate that all transitions (i.e., write->read, write->write, read->write, and read->read) among processing units (e.g. , from P0 to P1) are registered. However, transitions within the same processing block (for example, from P0 to P0) should not be registered. They do not need to be registered because they will be repeated through the normal execution of the thread of execution that was executing in that processing unit.

[00107] It should be understood that by using data such as those recorded in the above example, and with additional information from the CCP used by the processing device 102 on which the recording was made, the full ordering of operations that occurred in each thread of execution, and at least a partial ordering of operations among different processing units can be reconstructed. Thus, during the indexing process and/or through the reproduction of the trace file, equally, each of the above operations can be reconstructed, even if they were not all explicitly recorded in the trace file(s) 104d.

[00108] In some embodiments, the tracer 104a may write additional data packets to the trace file(s) 104d in order to improve the recording of ordering of operations among processing units. For example, tracer 104a may record some event ordering information such as monotonically incrementing number (MIN) (or some other counter/timer) to ensure that events that have a MIN (or other counter/timer) are fully ordered. ) among threads of execution. These MINs can be used to identify data packets corresponding to events that are defined as "orderable" among threads of execution. These events can be defined based on a "trace memory model" that defines how threads of execution can communicate through shared memory and how they share data in memory. As another example, tracer 104a may (periodically or randomly) record a hash value of the processor state based on a predetermined deterministic algorithm and a predetermined set of registers (eg, program counter, stack, general purpose registers, etc.). As another example, tracer 104a may (periodically or randomly) force cache line data to be logged. As yet another example, tracer 104a may trace "jump" packets that record a hash value of all or a portion (eg, a few bits) of the data they implicitly carry. Thus, when this implicit data is reconstructed at playback, appropriate portions of this implicit data can be hashed and matched against these jump packets to help identify their ordering. This can be useful, for example, if CCP cannot keep track of processor indexes associated with cache lines if the cache lines are in a shared state.

[00109] When tracer 104a writes additional data packets to trace file(s) 104d to improve orderliness, it is possible to skip recording some transitions among processing blocks. For example, it is possible to skip writing some read->read transitions among threads of execution. This can lead in some situations to "weakened" non-deterministic tracing because the ordering of some reads may not be fully reconstructable from the trace and CCP, but additional information about the ordering (e.g. MIN, processor state hash values, additional cache line data) can help reduce the scope of the search during playback to find a valid read order that does not "break" trace playback. The benefits of skipping some read->read transitions among threads of execution include the size of the trace and potentially simplified modifications to the processor 101 to facilitate the implementation of the trace.

[00110] FIG. 7A shows an example where some read->read transitions may be excluded from the trace depending on how processors are tracked. Like Fig. 6A, FIG. 7A includes a table 700a with a global ID of 701 and three columns (702a-702c) corresponding to three processing units (P0-P2). The omission of some read->read transitions is based on two observations. First, the entries must be in order; however, all reads between two consecutive writes (for example, reads by ID[3]-ID[7]) will read the same value, so the order among these reads does not matter (hence tracing with skipped read->read transitions). may be deterministic). Second, the presence of a "crossover" of reads and writes on replay (ie, reads and writes on the same cache line being replayed in the wrong order) means that the correct data is not being used for replay; however, having data (eg MIN, etc.) to avoid this error will help identify the actual ordering.

[00111] In the example shown in table 700a, the processing unit P2 only performs shared data reads, and these shared reads are only "captured" from other reads (eg, assuming that ID[9] left the cache line shared). If no registration entries are made for any of the read->read transitions (ie, ID[4]-ID[7] and ID[10]), there will be no information in the trace to correctly place the P2 reads. Based on the entries, it can be inferred that P2 never reads the value of DATA[1] (i.e., since the entry by ID[2] is not captured from P2), and in the absence of registration elements for P2's read->read transitions (i.e., i.e. ID[4], ID[7] and ID[10]), all that can be inferred for P2 is that there was at least one read from P2 between ID[2] and ID[8]. However, if there were registration entries for ID[4] and ID[10], the location of the remaining reads that may not need to be registered (i.e., ID[5]-ID[7] can be determined, as shown in Fig. 7B ). Each of these reads belongs to the same partition between records as the last registered read (ie, by ID[4]). Therefore, the location of these reads can be determined based on what writes are captured (and if there is no capture read operation, then there is no write after it until the next registered packet).

[00112] Considering table 700a, FIG. 7B shows an example of data logging skipping the read->read transitions highlighted in FIG. 7A that could be registered if "block bits" are used. Fig. 7C shows a table 700c that shows the logging of data that can be written if "index bits" are used and the indices are updated on reads.

[00113] As briefly mentioned above, some caches include both inclusive and exclusive levels (ie not fully inclusive cache). The registration methods described in this document apply to these caches as well as to fully inclusive or exclusive caches. As an example, FIG. 8A shows a computing environment 800a that includes two processors 801a/801b (eg, two processors in respective slots). Each processor 801 includes four processing units 802a/802b (eg, physical or logical processing units). Each processor 801 also includes a three-level cache including L1 layer 803a/803b, L2 layer 804a/804b, and L3 layer 805a/805b. As shown, each cache includes four L1 caches 803, each corresponding to one of the processing units 802. In addition, each cache includes two L2 caches 804, each corresponding to two of the processing units 802. In addition, each cache includes one L3 cache 805 for all processing units 802 in processor 801. Processing units and some of the caches are identified individually, for example, processors 802a of processor 801a are identified as A0-A3, L2 caches are identified as A4 and A5, and the L3 cache is identified as A6. Similar identifiers are used for the respective components in the processor 801b. Asterisks (*) associated with processing units A0, A1, A2, B0, and B1 indicate that registration is enabled for these processing units.

[00114] In computing environment 800a, caches may exhibit a combination of inclusive and exclusive behavior. For example, for the L3 cache A6 of the processor 801a, it may be inefficient to store a cache line if only processing unit A0 uses it. Instead, in such a case, the cache line may be stored in the L1 cache A0 and the L2 cache A4, but not in the L1 cache A1 or the L2 cache A5 or lower caches. To free up space, some caches may allow the A6 L3 cache to evict that cache line in such a situation. When this happens, A1 can get the cache line from A4's L2 cache, as would be normal in an inclusive cache. However, since the cache line does not exist in the L3 A6 cache or the L2 A5 cache, some cache implementations may also allow the cache line to side-shift, for example, from the L1 A0 cache to the L1 A2 or A3 caches. This can create some problems for tracing using CCP. The examples below show how tracing can be done using CCP in such situations.

[00115] FIG. 8B shows a table 800b that shows exemplary read and write operations performed by some processing units 802. The format of table 800b is similar to that of table 600a. Given computing environment 800a and table 800b, three different registration examples are now provided, each using different cache behaviors. These examples are described in the context of the following registration principles using CCP:

(1) Typically, data is logged when an address (cache line) goes from "unregistered" to "registered" (ie, based on determining that the cache line is participating in the registration, at step 304);

(2) Registration is generally discarded when the cache line goes from "registered" to "unregistered" or "crowded out" (although the registration will still be valid if that data is registered). However, this is valid for logging data evictions. This increases the size of the trace, but provides additional information that may help identify ordering among trace data streams, may help identify when trace playback is broken, and may provide additional trace analysis. As far as trace analysis is concerned, data eviction logging can provide more information about how the cache was used, can be used to identify performance characteristics of executable code, and can help pinpoint the time window for which a given cache line held a particular value. Embodiments for registering data wipes are discussed later with respect to FIG. 10A and 10B;

(3) Offset is detected when a cache line is shifted between cores or cache coherency status such that new information is provided;

(4) When a processing unit writes, invalidate the cache line for all other processing units. If the cache line has not already been registered for the processing unit, the system may either not register the cache line, or treat the write as: (i) a read (i.e. that registers the cache line and enables registration tracking), together with (ii) a write, assuming the memory in the cache line is readable by the processing unit. It may be acceptable, but less efficient, for the processor to disable logging and not log a record, but this will lose information that needs to be reconstructed on playback, and on average this may be inefficient in terms of trace size, since it is cheaper to register a link than register full data cache line later;

(5) It is reasonable to overregister (eg as in principle 2 above) to help with trace reconstruction later. Although this increases the size of the trace, it does not affect correctness. For example, some read->read transitions may be skipped (as described above with respect to FIGS. 7A-7C), but any internuclear transition that starts or ends with a write must be explicitly or implicitly registered. In another example, embodiments may add additional data packets (eg, which provide additional ordering information and/or hash values) to the trace at any time. In yet another example, embodiments may register when a cache line is first written after its CCP transition to the write state (i.e., a later hypothetical execution may cause the cache line to enter the write state, but not actually commit any or write to it). Logging these entries can make it easier to spread trace streams across cores later. In yet another example, embodiments may log indirect hops or other information that helps to quickly reduce the search area by separating trace data streams; and

(6) Incomplete registration (i.e. registration without all registered transitions) can still be used to reproduce the trace. This may result in additional computational overhead during playback to calculate the missing fragments.

[00116] In the first example shown in FIG. 8C, the CCP keeps track of the cache line status for each processing unit (ie, each core has its own read and write status). In this example, the cache behaves much like an inclusive cache, except that there may be data that moves through caches or through slots that are not available at the time of registration. For brevity, in these examples, processors 802 are referred to as "cores" and processors 801a and 801b are referred to as processors A and B or slots A and B. Additionally, a simplified registration notation of the form "ID:Core:From:Transition (i.e. from->to)" is used to represent the types of data that can be registered. This notation is explained in more detail in turn. For the first example, the registration might include:

[00117] By ID[0], "0:A0:R[DATA]->[1]", i.e., by ID[0], it is registered that the A0 core reads DATA[1], according to the aforementioned principle 1.

[00118] By ID[1], "1:B0:R[DATA]->[1]", i.e., by ID[1], it is registered that the B0 core reads DATA[1], also according to with the above principle 1. If the cache in processor B does not know that A0 has already registered data, then processor B registers it itself. Alternatively, if the cache at processor B knows that A0 has registered DATA[1], then the registration element may include "1:B0:R[A0]->R".

[00119] By ID[2], "2:A1:R[A0]->R", i.e., by ID[2], it is recorded that core A1 has made a read->read transition, and that A0 had access. Since the state of the cache line is shared by processor B, the entry may be of the form "2:A1:R:[A0,B0]->R", i.e., by ID[2], it is registered that core A1 has made a read branch ->read, and that A0 and B0 had access. Since traversing nests is usually more expensive than registering within a nest, the first registration element may be preferred for read->read transitions. However, when registering records to/from that go through sockets, the registration also goes through sockets.

[00120] According to ID[3], some embodiments do not register anything. Alternatively, since core A2 hasn't registered anything yet and the first thing it does is write, this can be registered as read->read. In any case, since the write has occurred, its cache line state of the other cores is invalidated. The cost (e.g. based on trace data) of logging a read->write on ID[3] will typically be less than logging the actual data on ID[4], so it may be advantageous to log at this stage. In this case, the registration element may include "3:A2:R[A0,B1,B0]->W", i.e. core A2 made a read->write transition, and cores A0, B1, and B0 had access.

[00121] What happens on ID[4] depends on what was registered on ID[3]. If nothing has been registered for ID[3], then data is registered (ie "4:A2:R[DATA]->[2]"). On the other hand, if the package was registered by ID[3], then there is nothing to register.

[00122] ID[5] has a read that goes through the cores. However, if core A2 still has the cache line modified (or equivalent), then the cache line serves the request (it cannot be served from memory). In this case, slot B will know that it came from slot A, and re-logging of data can be avoided; this can be logged as "5:B0:W[A2]->R". If the cache received data from main memory (this could be the case if slot A was able to update main memory and share its cache coherence state for the line), then the entry could be "5:B0:R[DATA]->2".

[00123] By ID[6], the operation is a normal read. Similar to reading by ID[2], slot B may or may not be aware of the data in slot A. If yes, the registration element may include "6:B1:R[B0,A2]->R"; otherwise, it may include "6:B1:R[B0]->R".

[00124] Per ID[7], if the cache line for B0 has not been evicted, there is nothing to register. If it has been evicted, processor B registers the data as coming from another core, or registers cache line data. This preemption of one core but not others in the socket generally does not occur in fully inclusive caches. In a fully inclusive cache, if any core in a socket has a cache line in its L1 cache, then L3 has a cache line, so a cache line cannot be evicted for one core but not another.

[00125] By ID[8], since core A0 has not registered anything since then, and the first operation to register is a write, this is similar to the operation by ID[3]. Processor A can log this as a read->write; alternatively, but perhaps less preferably, processor A may register nothing. If the packet is registered, its contents will vary depending on whether slot A can see slot B. If not, the packet MAY include "8:A0:R[A2]->W", and if possible, the packet MAY include "8:A0:R[B0,B1,A2]->W".

[00126] There is nothing to log on ID[9] if the package was registered on ID[8] (since it is an entry in an already registered cache), although the state of the cache line for other cores is usually invalidated if it was not already.

[00127] By ID[10] registration depends on what was registered by ID[8]. If no data has been registered on ID[8], then it must be done at this stage, so the packet may include "10:A1:R[DATA]->[4]". If the packet was registered by ID[8], this is a normal write>read packet (eg "10:A1:W[A0]->R").

[00128] ID[11] registers a read->read transition. If the package was registered by ID[8], then A0 is in the original kernel list (for example, "11:A2:R[A0,A1]->R"); otherwise A0 is not in the list (eg "11:A2:R[A1]->R").

[00129] According to ID[12], if socket B can see socket A, this is a read->read packet (eg, "12:B0:R[A0,A1,A2]->R"). If it cannot, then it is a complete data logging (eg "12:B0:R[DATA]->[4]").

[00130] At ID[13], data comes from B0, as well as socket A if visible (eg, "13:B1:R[A0,A1,A2,B0]->R"). The A0 core may be omitted from the list if the entry was not registered by ID[8].

[00131] At ID[14], nothing should be registered if the package registered at ID[8] has indeed already been registered. Otherwise, A0 will receive data from A1 and A2, and possibly socket B if it can be seen. Thus, the packet may include "14:A0:R[A1,A2,B0,B1]->R".

[00132] It should be noted that although the sockets were registered together in this example, it would be correct to register each socket separately, similar to how threads of execution can register separately. This can lead to larger traces, but has the advantage that there is no need to change any communication mechanism between sockets in the processor.

[00133] In addition, at any point in time, the cache line may be evicted, which would mean that the data must be collected from another core or re-registered. For example, if A0 had a cache line evicted before ID[11], then A2 will get the value from A1. If both A1 and A0 have been evicted, then processor A may need to log the value of the cache line into the trace for A2.

[00134] Finally, some processors may know that data is coming from another socket, but not from which core in that socket. In these cases, the processor may log the seniority (source) as the slot ID, log the data itself, or log the slot ID and hash value of the data (i.e., to help order accesses between slots, but should not log all of the data into the trace) .

[00135] In the second example shown in FIG. 8D, CCP uses indexes instead of keeping track of cache coherency on a per-core basis. In this environment, the index can be tracked between nests or within a nest. Given the performance comparison of inter-socket and intra-socket communications, the latter case (inside the nest) may be more rational. When an index is tracked within a socket, the trace may need to log something when data is shifted between sockets. This may include registering an index from another socket (but this may not necessarily be unambiguously sufficient for a deterministic trace), registering a hash value of one or more parts of a cache line value, or registering a packet in the sending socket trace to indicate that the data was sent.

[00136] When kernel indexes are tracked, when using a cache that is not fully inclusive, a complication arises when the L1 cache may have data that is not in the L3 cache. So, for example, imagine the following sequence of events: (i) A0 receives a row (and thus the index bits refer to A0) into its L1 cache; (ii) A1 receives the string (and hence the index bits refer to A1) into its L1 cache; (iii) the L3 cache flushes out the line; (iv) A1 evicts the line from its L1 cache; and (v) A2 gets the cache line from A0 into its L1 cache. The bottom line is that although A2 gets the cache line from A0, the index does not belong to A0. This makes it difficult to register mappings in a trace. Some solutions may include adding additional information (as described above) such as a hash value of one or more pieces of cache line data, periodically adding redundant information like a hash value of general purpose registers, and so on. Logging data evictions can also help, but it can significantly increase the size of the trace file and make logging more difficult (for example, logging data evictions from the L1 cache that are not in the L2 or L3 caches, but not logging data evictions from the L1 cache that are in the L2 caches or L3).

[00137] In some embodiments, when data is shifted from the L3 cache to child L2 or L1 caches, the registration entry is only inserted when the index changes. For example, suppose A0 had the line in its L1 cache (so the index bits refer to A0), then A1 gets the line into its L1 cache (index in zone A1), then both pop the cache line, but the shared L2 (or L3) still has it. If the L2 cache serves A1, then there is nothing to register. If the L2 cache serves A0, then no registration entry is required if it is known that A0 already had data; but if it is not known (or cannot be determined) whether A0 already had data, then the processor may need to register read->read.

[00138] Table 800d represents the log of operations from table 800b, provided that slots are logged independently, that tracking is done by index, that there are no additional latent data evictions, and that all writes that affect CCP and that occur when logging is enabled are logged. (for example, one entry must be registered if subsequent entries are from the same kernel and there is no access between entries from another kernel or other external object). For the second example, the registration might include:

[00139] For ID[0], "0:A0:R[DATA]->[1]".

[00140] For ID[1], "1:B0:R[DATA]->[1]", i.e. recall that each nest is registered separately.

[00141] For ID[2], "2:A1:R [A0]->R".

[00142] For ID[3], "3:A2:R [A1]->W".

[00143] For ID[4], nothing.

[00144] For ID[5], "5:B0:R[DATA]->[2]". This is because writing to ID[3] invalidates the row across all slots, and the slots are traced independently (as above).

[00145] For ID[6], "6:B1:R[B0]->R".

[00146] For ID[7], if the cache line for B0 has not been evicted, there is nothing to register.

[00147] For ID[8]: "8:A0:R[A2]->W", as bit registration is in progress (and despite this, the kernel did not register data before). This element demonstrates why, when using indexes, only the last owner of the socket is known.

[00148] There is nothing to register for ID[9].

[00149] For ID[10], "10:A1:W[A0]->R".

[00150] For ID[11], "11:A2:R[A1]->R".

[00151] For ID[12], "12:B0:R[data]->[4]". This is because the cache line has been invalidated across all sockets by ID[8].

[00152] For ID[13], "13:B1:R[B0]->R".

[00153] For ID[14], "14:A0:R[A2]->R". Note that for ID[11], the index has been updated to A2. Also note that it would not be known that this core already had data (i.e. ID[9]), since the index does not carry this information, whereas previously the state per processor (block bits) could carry information.

[00154] In a third example, the caches in environment 800a are unable to keep track of which core has the last shared (read) access to a cache line. Thus, in this example, the index of the last reader cannot be tracked since there are no bits for that. In this situation, the CCP can use one index value (which is not mapped to any core) to signal a shared string, another index value to signal an invalid string, and a handler index for the "modified" state (for example, using the MSI protocol ). In this third example, the registration may include registering a cache index in the package instead of a kernel index. An ancestor-to-child offset need not be logged, but may be logged as additional data. If an ancestor-to-child offset is not registered, it may be necessary to provide a parent-child hierarchy of caches in order to interpret the registration.

[00155] As mentioned above, in some environments, each cache line in the cache may include one flag bit, but the processor CCP may keep track of the coherence state for each cache line, with reference to the index for the processing unit to which the cache line's coherence state belongs. . As already mentioned, this creates completely deterministic traces, but can lead to larger traces than in cases where there is information on each processing unit (for example, CCP, which traces each processing unit, in combination with a flag bit per cache line) . Fig. 9A and 9B show how registration can differ in these two situations (ie, CCP block information plus cache line flag bit versus CCP index plus cache line flag bit). Fig. 9A shows a table 900a that shows reads and writes by two processing units (P0 and P1), and FIG. 9B shows a table 900b comparing when registration items can be made in the two environments. In these examples, assume that the flag bit starts in a cleared state, and that the block bits/index bits indicate that no processing block has access to the cache line.

[00156] Initially, if the CCP keeps track of block information and the cache line uses a flag bit, registration may proceed as follows. As shown in table 900b, nothing needs to be logged on ID[0] since it is a cache line entry that has not been registered (alternatively, the value before the write could have been registered and the flag bit could have been set). At this point, the CCP may note that neither P0 nor P1 has access to the cache line. By ID[1] cache line data can be registered for P1. The flag bit may be turned on and the CCP may indicate that P1 has access to the cache line. On ID[2], a read->read packet may be logged, with P0 taking the cache line from P1 (this is logged because the flag bit was turned on and CCP is used to determine that P0 has no access). The flag bit has already been turned on, and CCP notes that P0 now also has access to the state of the cache line. Nothing needs to be registered for ID[3] (the cache line is already in the registration for this kernel). This is determined because the flag bit is on and the CCP indicates that P1 has already accessed the cache line. By ID[4] a read->write packet for P0 can be registered. This is because the flag bit is on and P0 has already accessed the cache line. Because it was a write, CCP can invalidate the cache line for all other processing devices (ie, P0 has access, but P1 does not). By ID[5] a write->read packet for P1 can be registered. This is because the flag bit is on but P1 has data in the trace (as specified by CCP). It should be noted that two reference packets on ID[4] and ID[5] are less than empty registration on ID[4] and the need to register data on ID[5] thereafter. CCP notes that P1 now has access to the cache line, in addition to P0.

[00157] Now, if the CCP only keeps track of index information and the cache line uses a flag bit, registration can proceed as follows. As shown in table 900b, nothing needs to be logged on ID[0] since the flag bit is off and this is a record. As before, this can alternatively be logged as a read plus a write if the memory is read from the P0 side. By ID[1] cache line data can be registered for P1. The flag bit may be turned on, and the CCP and index update must point to P1. By ID[2], a read->read packet for P0 can be registered. This is because the flag bit is already on and the index is pointing to P1. CCP can update the index to P0. This packet ID[3] read->read can be registered for P1. Note that this case is now indistinguishable from ID[2] because in both cases the index is directed to a different processor, the flag bit is on, and the cache line is in a shared state. CCP may update the index to P1. By ID[4] a read->write packet for P0 can be registered. The flag bit is on, which means the packet can be registered by reference. This updates the CCP index to P0. Packet write->read by ID[5] can be registered for P1. The flag bit is on, the packet is registered by reference. The cache line becomes shared, so CCP updates the index to P1. As shown in table 900b, the index option results in a larger trace file than the block option, but still produces a fully deterministic trace.

[00158] Some embodiments herein indicate that, in terms of trace file size, it may be advantageous to write data packets that refer to data owned by another processing unit (when possible) rather than writing cache line data later (for example, ID[4] in each of the previous examples). Other benefits can also be gained from posting by reference. For example, during playback, when a number of registration elements that are referential, it can be concluded that there was no external interference with the cache line data. This is because when the full cache line data is re-registered, it means that the cache line has either been evicted or invalidated. Thus, including registration elements by reference, even in a situation where the registration element may not be strictly required, can provide implicit information about the absence of external interference, which can be useful information for playback or for debugging.

[00159] In some implementations, addresses that are registered in trace elements (eg, the aforementioned "@" elements) contain physical memory addresses. In these implementations, processor 102 may write one or more TLB entries 102f to trace file(s) 104d. This may be as part of the trace data streams for different processing units, or as part of additional trace data streams. This will allow the playback software to later map these physical addresses to virtual addresses.

[00160] In addition, because physical addresses can sometimes be considered "secret" information (eg, when written at the user mode level), some embodiments record some representation of the actual physical addresses rather than the physical addresses themselves. This representation can be any representation that uniquely maps its identifiers to physical addresses without exposing the physical address. One example would be the hash value of each physical address. When these representations are used and TLB entries 102f are written to trace file(s) 104d, processor 102 writes a mapping between these representations and virtual addresses, rather than physical addresses to virtual addresses.

[00161] As mentioned, processor 102 may include one or more buffers 102e. These buffers may be used as a temporary storage location for trace file elements, prior to actually writing those elements to trace file(s) 102f. Thus, when step 305 results in logging data into a trace, step 305 may include a step that writes data to buffer(s) 102e. In some embodiments, processor 102 employs lazy logging techniques to reduce the impact of writing trace data on processor 102 and the memory bus. In these embodiments, processor 102 may store trace data in buffer(s) 102e and defer writing to trace file(s) 102f until memory bus bandwidth is available or buffer(s) 102e are full. .

[00162] As also mentioned, some embodiments may log cache evictions. Fig. 10A and 10B show some implementations in which cache eviction can be effectively logged (ie, in terms of trace file size) by maximizing the properties of associative caches. Initially, in Fig. 10A shows an example of 1000 different parts of a memory address, and their relationship to associative caches. As shown, memory addresses include a first set of bits 1001 which are the least significant bits of the address and which are typically zero. The first set of bits 1001 is zero because memory addresses are normally aligned to the size of the memory address (eg, 32 bits, 64 bits, etc.). Thus, the number of the first set of bits 1001 depends on the size of the memory address. For example, if the memory address is 32 bits (i.e. 2^5 bits), then the first bit set 1001 contains five bits (so that the memory addresses are a multiple of 32), if the memory address is 64 bits (i.e. 2 ^6), then the first set of bits 1001 contains six bits (so that the memory addresses are a multiple of 64), and so on. The memory addresses also include a second set of bits 1002 that can be used by the processor 102 to determine the particular group of addresses in the associative cache where the memory address data is to be stored. In example 1000 in FIG. 10A, for example, the second set of bits 1002 contains three bits that would correspond to an associative cache having eight address groups. The number of the second set of bits 1002 therefore depends on the specific associative cache geometry. The memory addresses also include a third set of bits 1003 containing the remaining MSBs of the memory address.

[00163] Considering FIG. 10A, FIG. 10B shows an example 1004 of registering cache misses and cache evictions in an associative cache. Initially, example 1004 shows three memory addresses 1005 (ie address 1024), 1006 (ie address @2112), and 1007 (ie address @2048). Fig. 10B also shows an associative cache 1010 that has eight groups, each containing four entries. The binary identity of these groups and entries is shown in columns 1008 (groups) and 1009 (entries), along with the corresponding decimal representation in parentheses. Thus, for example, the (0,0) cache line, i.e. group 0, entry 0, in cache 1010 is shown in binary as group '000' (column 1008) and entry '00' (column 1009); cache line (0,1), group 0, entry 1, in cache 1010 is shown in binary as group '000' (column 1008) and entry '01' (column 1009); and so on up to the cache line (8,3), i.e. group 8, entry 3, in cache 1010, which is shown in binary as group '111' (column 1008) and entry '11' (column 1009).

[00164] Now suppose there is a first cache miss at address 1005 (ie, @1024). Here, since its second set of bits 1002 is '000', processor 102 may determine that it should store the data corresponding to address 1005 in group 0 of cache 1010. The particular entry in group 0 is typically selected through processor-specific logic. With reference to example 1004, however, assume that data is stored in input 0 (as indicated by the arrow) 1011a. In connection with this cache miss, the log data written by tracer 104a may include the memory address (ie, @1024) and the entry (ie, entry 0) in which the data was stored. Note that any number of compression methods can be used to reduce the number of bits required to store a memory address in a trace. The group (ie, group 0) does not need to be registered because it can be obtained from the second set of bits 1002 of the memory address.

[00165] Next, assume that there is a second cache miss at address 1006 (ie, @2112). This time, the second set of bits 1002 is '010', so processor 102 can determine that it should store the data corresponding to address 1006 in group 2 of cache 1010. Again, a particular entry in group 2 is typically selected by a processor-specific logic. With reference to example 1004, however, assume that data is stored in input 0 (as indicated by the arrow) 1011b. In connection with this cache miss, the log data written by tracer 104a may include the memory address (ie, @2112) and the entry (ie, entry 0) in which the data was stored. Again, the group (ie, group 2) does not need to be registered because it can be obtained from the second set of bits 1002 of the memory address.

[00166] Now suppose there is a third cache miss at address 1007 (ie, @2048). The second set of bits 1002 is again '000', so processor 102 may determine that it should store the data corresponding to address 1007 in group 0 of cache 1010. The particular entry is again selected through processor-specific logic, but assume that processor selected input 0 (as shown by arrow 1011c). In connection with this cache miss, the log data written by tracer 104a may include the memory address (ie, @2048) and the entry (ie, path 0) in which the data was stored. Again, the group (ie, group 0) does not need to be registered because it can be obtained from the second set of bits 1002 of the memory address.

[00167] Since this cache line (0,0) currently corresponds to address 1005, this third cache miss at address 1007 causes address 1005 to be evicted from cache 1010. However, embodiments may refuse to write any trace data documenting it's data preemption. This is because data preemption can be inferred from data already in the trace, i.e., the first cache miss at address 1005 to entry 0 combined with the second cache miss at address 1007 to entry 0. Even though that the group (ie, group 0) may not be explicitly logged in the trace, it may be inferred from these addresses. Accordingly, replaying this trace data may repeat the data extrusion.

[00168] Some data evictions are the result of events other than a cache miss. For example, CCP may cause data preemption to occur in order to maintain consistency across different caches. Suppose, for example, that address 1006 is evicted from cache line (2,0) in cache 1010 due to a CCP event. At this stage, the data preemption can be explicitly registered by writing a group (ie, '010') and an entry (ie, '00') of the data preemption. Significantly, the address that has been evicted should not be logged, as it was already logged when the second cache miss was logged, which brought address 1006 into line (2,0) of the cache. Accordingly, in this example, data preemption can be completely captured in trace file(s) 104d with as little as five bits of log data (before any form of compression).

[00169] Some embodiments are also capable of safely tracing the actions of a processing unit, even when execution of a thread of execution in that processing unit interacts with the secure enclave. As will be appreciated by those skilled in the art, enclaves are hardware-based security that can protect sensitive information (e.g., cryptographic keys, credentials, biometric data, etc.), potentially even from software execution on processing device 102. providing the lowest level. Thus, in addition to protecting sensitive information from user-mode processes, enclaves can even protect sensitive information from kernels and/or hypervisors. In many implementations, enclaves are presented to the executing process as an encrypted piece(s) of memory mapped to the process's address space. This can be done, for example, by using different page tables for the executing process and the enclave. When a process interacts with an enclave, the process can read/write to its own mapped memory, and the enclave can read/write to its own mapped memory and/or the process mapped memory.

[00170] First embodiments of enclave-aware tracing trace an executing process while refraining from tracing the enclave with which the process interacts, while still allowing full replay of the process being traced. In these embodiments, memory reads by an executable process into its address space are traced/logged using one or more of the mechanisms already described herein. However, when a context switch to the enclave takes place, embodiments may track any memory locations that were previously read by the traced process and that were written by the enclave during its execution. When the tracee runs again after switching to the enclave, these memory locations are considered unregistered by the traced process. Thus, if the process being traced reads from those memory locations again (potentially reading data that was placed in those locations by the enclave), those reads are logged into the trace. In essence, this means that any side effects of enclave execution that are visible to the traced process are captured in the trace without the enclave's execution being required to be traced. Thus, the traceable process can be replayed later with these side effects, without actually needing (or even being able to) replay the execution of the enclave. There are several mechanisms (described earlier) that can be used to keep track of memory locations that were previously read by the traced process and that were written by the enclave during its execution, for example, accounting bits (for example, flag bits, block bits, index bits), locking login, use of CCP data, etc.

[00171] Second embodiments of enclave-aware tracing trace an executable process (e.g., based on access, such as reads, to its own address space) while tracing the enclave (e.g., based on access to its own address space and/or access to the address space of the traced process). These embodiments can be implemented when there is a required level of trust between the kernel/hypervisor and the enclave. In these embodiments, the enclave execution trace data may be logged into a separate trace data stream and/or encrypted such that no replay entity can reproduce the enclave without access to the enclave's separate trace data stream and/or the cryptographic key( sam) that can be used to decipher trace data related to enclave execution.

[00172] Third embodiments of enclave-aware tracing combine the first and second embodiments. Thus, these third embodiments can record a trace of an executing process that includes the side effects of that process's use of the enclave (i.e., the first embodiments), along with a trace of the enclave itself (i.e., the second embodiments). This allows a traceable process to be replayed by a user who does not have the required privilege level and/or cryptographic key(s), while allowing a user who has the required privilege level and/or cryptographic key(s) to also replay the execution of the enclave itself.

[00173] Each of these enclave tracing embodiments is applicable without enclaves, and to any situation in which the traceable object interacts with another object whose execution must be protected during the trace (referred to in these circumstances as a protected object). For example, any of these embodiments can be used when tracing a user-mode process that is interacting with a kernel-mode process, in which case the kernel-mode process can be treated much like an enclave. In another example, either of these embodiments could be used when tracing a kernel-mode process that interacts with a hypervisor, in which case the hypervisor could be considered much the same as an enclave.

[00174] There may be environments in which it is impractical (for example, for performance or security reasons), impossible (for example, due to lack of hardware support), or undesirable to keep track of which memory cells that were previously read by the traced process are written by the protected object during its execution. This may preclude the use of the enclave tracing embodiments described above. However, there are also methods for tracing in these situations as well.

[00175] The first method is to treat the processor's cache as invalid after a context switch away from the protected object. Treating the processor's cache as invalid causes reads by the traceable object after returning from the protected object to result in cache misses that may be logged. These cache misses will include any values that have been modified by the protected object in the address space of the traceable object and that have subsequently been read by the traceable object. While this method may generate more trace data than the three embodiments described above, it does capture the execution effects of the protected object that the traced object depended on. In some embodiments, this first method may also write one or more keyframes (eg, including a snapshot of the processor's registers) after returning to the traceable object from the protected object. The key frame(s) allow playback of the traced object to begin upon return from the protected object, even though there is no continuity in the trace data (ie, during the execution of the protected object).

[00176] The second method is to log cache misses related to reads by the protected object from the address space of the traceable object, as well as writes made by the protected object to the address space of the traceable object. This allows traces to be replayed for repeated writes of a protected object without having to access the instructions of the protected object that produced them. It also gives replay access to data (in the tracee's address space) that the protected object read and later accessed by the tracee. Mixed approaches are possible (if there is sufficient overhead, such as CCP data), whereby writes of the protected object (in the address space of the traceable object) may be logged, but reads of the protected object may not be logged if those reads are logged later due to cache invalidation.

[00177] The present invention may be embodied in other specific forms without departing from its spirit or essential properties. The described embodiments are to be considered in all respects only as illustrative and not restrictive. Therefore, the scope of the present invention is indicated by the appended claims and not by the preceding description. All changes that are made within the meaning and range of equivalence of the claims are to be construed as being within the scope of the claims.

Claims (43)

1. Computing device containing: a plurality of processing units; cache memory containing a plurality of cache lines that are used to cache data from one or more backing stores and that are shared by multiple processing units, and the consistency of data in this set of cache lines and said one or more backing stores is regulated according to the Cache Coherence Protocol (CCP ); and stored control logic that configures the computing device to do at least the following: determining that at least the following conditions are met: (i) the operation caused an interaction between a particular cache line from said plurality of cache lines and said one or more backing stores; (ii) registration is enabled for the particular processing unit of said plurality of processing units that caused the operation; (iii) said particular cache line is a member in the registration; and (iv) CCP indicates that there is data to be logged in the trace based on said operation; and on the basis of at least a determination that said conditions are met, allowing the data to be logged in the trace, wherein the data can be used to reproduce said operation. 2. The computing device of claim 1, wherein the stored control logic also configures the computing device to update one or more accounting bits associated with said particular cache line to indicate whether that particular cache line remains a member in the registration after said operation. 3. The computing device of claim 2, wherein said one or more accounting bits associated with a particular cache line comprise one of the following: (i) a single bit, (ii) a plurality of bits, each corresponding to one of said plurality of blocks processing, and (iii) a set of bits that store the index value of the processing device. 4. The computing device of claim 2, wherein said one or more accounting bits associated with a particular cache line are stored in one or more reserved cache lines that are separate from cache lines that are used to cache data from one or more backing stores. . 5. The computing device of claim 1, wherein providing data logging to the trace comprises writing data to a buffer, wherein flushing data from the buffer to the trace file is delayed based on memory bus activity. 6. The computing device of claim 1, wherein the stored control logic also configures the computing device to register at least one cache eviction by reference to a group and an entry in the associative cache. 7. The computing device according to claim 1, wherein the logged data contains transitions between different states of the CCP. 8. The computing device of claim 1, wherein the logged data comprises at least one of the following: a transition from a write state to a read state, a transition from a write state to a write state, and a transition from a read state to a write state. 9. The computing device of claim 1, wherein using the CCP to identify that there is data to be logged in the trace comprises identifying that it is not required that the transition from the read state to the read state be logged in the trace. 10. The computing device of claim 1, wherein the data for each processing unit is logged into at least one separate data stream. 11. The computing device of claim 1, wherein the data for the two or more processing units is logged into the same data stream but is tagged with a processing unit identifier. 12. The computing device according to claim 1, wherein the data to be recorded in the trace contains ordering information. 13. The computing device according to claim 1, wherein the data to be logged comprises data written to said specific cache line by the enclave, wherein the provision for logging data in the trace comprises: when said operation that caused an interaction between a particular cache line and one or more backing stores corresponds to a thread of execution interacting with the enclave, ensuring that the data is logged to the trace data stream corresponding to that thread of execution; or when said operation that caused an interaction between a particular cache line and one or more backing stores matches an enclave, ensuring data logging is separate from the trace data stream corresponding to that thread of execution. 14. A method implemented in a computing environment that includes a plurality of processing units and a cache memory containing a plurality of cache lines that are used to cache data from one or more backing stores and that are shared by multiple processing units, and data consistency in this the plurality of cache lines and said one or more backing stores is governed by the Cache Coherence Protocol, wherein the method is for performing a cache-based trace recording using Cache Coherence Protocol (CPP) data and comprising the steps of: determine that at least the following conditions are met: (i) the operation caused an interaction between a particular cache line from said plurality of cache lines and said one or more backing stores; (ii) registration is enabled for the particular processing unit of said plurality of processing units that caused the operation; (iii) said particular cache line is a member in the registration; and (iv) CCP indicates that there is data to be logged in the trace based on said operation; and on the basis of at least determining that said conditions are met, data is logged in the trace, wherein the data can be used to reproduce said operation. 15. The method of claim 14, further comprising updating one or more accounting bits associated with said particular cache line to indicate whether that particular cache line remains a member in the registration after said operation. 16. The method of claim 14, wherein providing data logging to the trace comprises writing data to a buffer, wherein flushing the data from the buffer to the trace file is deferred based on memory bus activity. 17. The method of claim 14, wherein the logged data comprises transitions between different CCP states. 18. The method of claim 14, wherein the logged data comprises at least one of a transition from a write state to a read state, a transition from a write state to a write state, and a transition from a read state to a write state. 19. The method of claim 14, wherein using the CCP to identify that there is data to be logged in the trace comprises identifying that the read-to-read state transition is not required to be logged in the trace. 20. A computer-readable storage medium that stores computer-executable instructions for use in a computing device that contains a plurality of processing units and a cache memory containing a plurality of cache lines that are used to cache data from one or more backing stores and that are shared by a plurality of processing units , wherein the consistency of the data in that plurality of cache lines and said one or more backing stores is governed by the Cache Coherence Protocol (CCP), wherein the computer-executable instructions are executable by one or more processing units to cause the computing device to do at least the following: determining that at least the following conditions are met: (i) the operation caused an interaction between a particular cache line from said plurality of cache lines and said one or more backing stores; (ii) registration is enabled for the particular processing unit of said plurality of processing units that caused the operation; (iii) said particular cache line is a member in the registration; and (iv) CCP indicates that there is data to be logged in the trace based on said operation; and on the basis of at least a determination that said conditions are met, allowing the data to be logged in the trace, wherein the data can be used to reproduce said operation.

Images