GB2464179A - Identifying the source of a memory leak by grouping repeated patterns of the same object type - Google Patents

Abstract

Repeated patterns of the same object type are identified in a collection of dynamic data structures, such as a Java® object heap. The objects are then grouped together in a flat data structure and analysed to using a memory leak detection algorithm. The objects that are identified may form a linked list and some of the nodes may be ancestors of other nodes. The sizes of the nodes may be calculated and a memory leak may be detected by finding a child node which is smaller than its parent. Alternatively, the number of child nodes may be calculated and a memory leak identified if a node has a large number of child nodes. The objects may be in a memory image.

Description

DETECTING MEMORY LEAKS IN DYNAMIC DATA COLLECTIONS

This invention relates to the field of data processing. In particular, the invention relates to monitoring program code memory use and detecting memory leaks in dynamic data collections.

Object oriented programming employs collections of objects or discrete modular data structures that are identified by references. The references can be stored in application variables and within the objects forming a network or objects and references, known as the reference graph. The objects are created dynamically during the application execution and are contained in a memory structure referred to as a heap.

Many object orientated programming languages employ automatic memory management.

Automatic memory management is an active component of the runtime system associated with the implementation of the object oriented language, which removes unneeded objects from the heap during the application execution. An object is unneeded if the application will no longer use it during its execution.

A problem with automatic memory management is that some objects due to design or coding errors remain live, contrary to the programmer's intentions. These objects, referred to as lingering objects, accumulate over time, clogging the heap and causing performance problems.

To detect lingering objects, programmers in the development phase of the application employ memory debugging or memory profiling tools.

There are currently several tools that carry out the analysis of objects stored in the memory image of a Java process (Java is a trade mark of Sun Microsystems, Inc.). For example, HeapAnalyzer and MDD4J by IBM (HeapAnalyzer, MDD4J and IBM are trade marks of International Business Machines Corporation), Memory Analysis Tool by SAP AG, and Heap Analysis Tool by Sun Microsystems, Inc. Some of these memory analysis tools, such as HeapAnalyzer, MDD4J and Memory Analysis Tool, can also carry out memory leak analysis by identifying large data collections. This is a function that is seen as hugely important in the analysis of OutOfMemoryErrors from Java applications which is the highest occurring Java problem scenario.

These tools are limited in the detection of large collections by the algorithms that they currently employ: they are capable of finding "flat" (array type) data collections, but are not effective at finding "deep" (linked list type) data collections.

United States Patent Application No. 2006/02 12852 describes a method of detecting memory leaks in which a Depth First Search (DFS) is carried out to determine the memory "reach size" from any given node in the object graph. Analysis is then carried out using the following two mechanisms to identify large collections: 1. A large drop in total reach size from a parent node to a child node; and 2. A parent node that has a large number of child nodes This has the limitation of not being able to detect "deep" (linked list type) collections as the drop in reach size and the number of child nodes is spread over a number of entries in the "deep" collection.

According to a first aspect of the present invention there is provided a method for detecting a memory leak, comprising: identifying repeated patterns of a same object type in a dynamic data structure collection; grouping the identified objects in a resultant flat data structure; and applying a memory leak algorithm to the resultant flat data structure.

According to a second aspect of the present invention there is provided a computer program product stored on a computer readable storage medium for detecting a memory leak, comprising computer readable program code means for performing the steps of: identifying repeated patterns of a same object type in a dynamic data structure collection; grouping the identified objects in a resultant flat data structure; and applying a memory leak algorithm to the resultant flat data structure.

According to a third aspect of the present invention there is provided a method of providing a service to a customer over a network, the service comprising: identifying repeated patterns of a same object type in a dynamic data structure collection; grouping the identified objects in a resultant flat data structure; and applying a memory leak algorithm to the resultant flat data structure.

According to a fourth aspect of the present invention there is provided a system for detecting a memory leak, comprising: a processor; means for identifying repeated patterns of a same object type in a dynamic data structure collection; means for grouping the identified objects in a resultant flat data structure; and means for applying a memory leak algorithm to the resultant flat data structure.

The subject matter regarded as the invention is particularly pointed out and distinctly claimed in the concluding portion of the specification. The invention, both as to organization and method of operation, together with objects, features, and advantages thereof, may best be understood by reference to the following detailed description when read with the accompanying drawings in which: Figure 1 is a flow diagram of a method in accordance with the present invention; Figures 2A and 2B are schematic representations of objects in linked lists in accordance with an aspect of the present invention; Figures 3A and 3B are schematic representations of objects in linked lists in accordance with an aspect of the present invention; Figure 4 is block diagram of a system in accordance with the present invention; and Figure 5 is a block diagram of a computer system in which the present invention may be implemented.

It will be appreciated that for simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, the dimensions of some of the elements may be exaggerated relative to other elements for clarity. Further, where considered appropriate, reference numbers may be repeated among the figures to indicate corresponding or analogous features.

In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the present invention.

A method and system for memory leak detection in large dynamic data collections, such as linked list type collections, are described. A pre-processing step of identifying repeated patterns of same object types is applied to obtain a flat or imperative data collection to which known memory leak algorithms may be applied.

The pre-processing step identifies repeated patterns of same object types, such as ancestor nodes of parent, child, grandchild, etc., and grouping of all the same object type entries in the collection. Once this has been done, known algorithms for memory leak detection for flat data collections can be used for the deep collection as a whole. For example, the drop in reach size or a large number of child nodes algorithms.

When grouping the same object type entries, a "super node" may be formulated in the data structure. For example, the linked list entries are identified and the parent, child, grandchild etc. node merged. The entries in the linked list can be identified by searching for cases where parent and child nodes are of the same object type, which is one of the characteristics of linked list type collections.

Referring to Figure 1, a flow diagram 100 shows a general method of inputting 101 a dynamic data type collection, identifying 102 and grouping 103 repeated patterns of same object types in the entries of the dynamic data type collection. For example, the dynamic data type collection may be a linked list type collection including linked lists and doubly linked lists (with two links from parent to child, one in each direction). Java provides a LinkedList implementation; however, the described method may also be applied to user implementations that use the well understood concept of a linked list. The same object types may be repeated pattern may be ancestor nodes of the same object type such as a grandparent, parent, child, grandchild, etc. Optionally, the grouped repeated patterns of same object types are collapsed 104 or merged into a super node" to provide a virtualized object. However, this is not essential and the identified repeated pattern of same object types may be processed by the memory leak algorithm without virtualizing as a super node.

The resulting data collection with grouped repeated patterns of same object types has a known memory leak algorithm applied 105.

Existing memory leak detection algorithms are effective for imperative data structures that are allocated in contiguous memory (such as arrays, records, etc) but not for dynamic data structures (such as linked lists). The described method identifies dynamic data structures and represents them as imperative data structures so that memory leak algorithms can be effectively applied to them as a whole.

One known memory leak algorithm is described in US 2006/02 12852 which identifies large collections using a Depth First Search (DFS) to determine the memory "reach size" from any given node in the object graph. Analysis is carried out using a large drop in total reach size from a parent node to a child node and recognising a parent node that has a large number of child nodes.

This known DFS memory algorithm can be applied to the resulting data collection after the described pre-processing step with identified repeated patterns of same object types. A reach size for a super node can be calculated when merging the same object types in the repeated pattern.

For example, should an application contain a linked list created using the LinkedList class provided in the Java SE core application programming interface, each node of the linked list is of type "java/util/LinkedList$Entry". If this linked list has 100 entries, and all 100 entries are collapsed into a single "super node" representing the linked list, the list would look like a "flat" array-like data collection. The node would have 100 child nodes (the entry object from each part of the linked list), and the drop in reach size across the super node would be usable as the measure of the memory held by the entire linked list collection.

This would also apply for any applicationlcustom generated linked list implementations that have alternative type names.

Referring to Figures 2A and 2B, schematic representations are provided of collapsing a linked list into a virtualized "super node" object. Figure 2A shows a collection 200 of objects is provided including a first object X 201 and a plurality of objects LL1-LLn 211-216 in a linkedlist2l0.

The process identifies a linked list 210 because each of the items 211-216 in the linked list 210 is of the same type, for example, LL in the illustrated case. Each linked list entry 211- 216 (LL1, LL2 etc) references a single element/data object 221-226 El-En and references the next linked list object directly.

In another example embodiment, LL1 could be a repeated pattern, e.g., Al -* Bl -* Cl which then references A2 -* B2 -* B3, etc. Figure 2B shows a resultant flat data structure 250 with the linked list 210 collapsed into a virtualized linked list 230 after the first object X 201. The virtualized linked list 230 references each of the data objects 221-226 El-En.

Standard graph analysis techniques that are already used by memory analysis tools may be used to show a virtualized object. Each node has a value that is the size of the node, as well as a value to describe the total combined sizes of the nodes under it, which is known as the "reach size". A known analysis for large collections looks for a large drop in reach size between two nodes.

Once the linked list has been collapsed into the uvirtualized representation, the Virtualized Linked List has a size that is the sum total of the sizes of the nodes in the linked list and the drop in reach size is now large.

Figures 3A and 3B show the same schematic representations as Figures 2A and 2B annotating each node with its reach size and working in a simplistic case where each node is of size 1.

Starting from the end of the linked list 210 and working up it, node 216 has a value of 2: 1 for itself, and 1 for its child node 226. The next node 215 up in the linked list has a value of 4: 1 for itself, 1 for its child node 225, and 2 for node 216. The next node 214 up in the linked list has a value of 6: 1 for itself, I for its child node 224, and 4 for node 215. This continues up the linked list 210 to node 211 which has a value of 12: 1 for itself, 1 for its child node 221, and 10 for previous node 212.

This gives the Virtualized Linked List 230 a reach size of 12: 6 child nodes plus the size of the Virtualized Linked List 230 which is the sum of its parts, i.e. 6. There is now a large drop in reach size of 12 -1 between the Virtualized Linked List 230 and the size of the largest child node, in this case 1. Therefore, using an algorithm using a Depth First Search (DFS) to determine the memory "reach size", a large collection has been identified. The calculation is normally more complex as each node can have a different size.

In one embodiment, a memory leak algorithm is created for linked lists directly, using the same criteria as for generating the super node. Identifying the linked list using a repeated pattern of the same object/node type is used without virtualization representation with the drop in reach size calculation being made to span the linked list directly.

In another embodiment, representing the linked list as a virtualized linked list makes the analysis simpler using existing techniques and visualization easier. The use of virtualized representation of the Virtualized Link List both allows reuse of the existing leak detection algorithms and provides users of any graphical user interface (GUT) based tools the ability to view the Linked List type collection as a single virtualized object.

This second capability makes visualization of the collections in the Java heap much easier. A linked list is very "wide". A linked list that is of any real size (say 200+ entries) can not easily be displayed on a screen in its entirety (a user would have to scroll left/right to see the entire linked list). By using the virtualized linked list representation, the whole collection can be represented on-screen, and the data collection becomes easier to comprehend.

Referring to Figure 4 a block diagram shows an example system 400 using the described method of memory leak analysis.

The system 400 includes a runtime system 410 including an application 411 in an object oriented language. Data structures 420 are stored in the system including objects created dynamically during the application's 411 execution and contained in a memory structure referred to as a heap 422 and an application's 411 associated reference graph 421 of objects and references. The reference graph 421 defines parent/child relationships based on information from the heap 422.

An automatic memory manager 413 is provided which removes unneeded objects from the heap 422 during the application's 411 execution.

A memory leak analysis tool 430 is provided which may be executed on the system 400 or as a service via a network. The memory leak analysis tool 430 includes means for identifying and grouping repeated patterns of same object types 431, and means for applying a memory leak analysis algorithm 435.

The memory leak analysis tool 430 also, optionally, includes a means for merging 432 identified repeated patterns of same object types as a super node and means for graphically displaying 433 the super node in a GUI 434 of the memory leak analysis tool 430.

The memory leak analysis tool 430 also optionally includes means for determining 436 and storing the object size of the super node as the combined object sizes of the identified objects and means for representing 437 as child nodes the objects referenced by the identified objects.

The combined object sizes and the child nodes may be used in the memory leak analysis algorithm.

The memory leak analysis tool 430 may be used in any problem determination or application development tool looking at memory usage in object orientated runtime, such as Java memory usage.

Referring to Figure 5, an exemplary system for implementing the memory leak analysis tool includes a data processing system 500 suitable for storing and/or executing program code including at least one processor 501 coupled directly or indirectly to memory elements through a bus system 503. The memory elements can include local memory employed during actual execution of the program code, bulk storage, and cache memories which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during execution.

The memory elements may include system memory 502 in the form of read only memory (ROM) 504 and random access memory (RAM) 505. A basic input/output system (BIOS) 506 may be stored in ROM 504. System software 507 may be stored in RAM 505 including operating system software 508. Software applications 510 may also be stored in RAM 505.

The system 500 may also include a primary storage means 511 such as a magnetic hard disk drive and secondary storage means 512 such as a magnetic disc drive and an optical disc drive. The drives and their associated computer-readable media provide non-volatile storage of computer-executable instructions, data structures, program modules and other data for the system 500. Software applications may be stored on the primary and secondary storage means 511, 512 as well as the system memory 502.

The computing system 500 may operate in a networked environment using logical connections to one or more remote computers via a network adapter 516.

Input/output devices 513 can be coupled to the system either directly or through intervening I/O controllers. A user may enter commands and information into the system 500 through input devices such as a keyboard, pointing device, or other input devices (for example, microphone, joy stick, game pad, satellite dish, scanner, or the like). Output devices may include speakers, printers, etc. A display device 514 is also connected to system bus 503 via an interface, such as video adapter 515.

A problem determination or application development tool looking at memory usage in an orientated runtime may use the described method and system.

A memory leak or memory usage tool alone or as part of an application development tool or problem determination tool may be provided as a service to a customer over a network.

The invention can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment containing both hardware and software elements. In an embodiment, the invention is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc. The invention can take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer usable or computer readable medium can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus or device.

The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus or device) or a propagation medium. Examples of a computer-readable medium include a semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read only memory (ROM), a rigid magnetic disk and an optical disk. Current examples of optical disks include compact disk read only memory (CD-ROM), compact disk read/write (CD-R/W), and DVD.

Improvements and modifications can be made to the foregoing without departing from the scope of the present invention.

Claims (20)

1. CLAIMS1. A method for detecting a memory leak, comprising: identiflying repeated patterns of a same object type in a dynamic data structure collection; grouping the identified objects in a resultant flat data structure; and applying a memory leak algorithm to the resultant flat data structure.
2. 2. The method as claimed in claim 1, wherein the dynamic data structure collection is an object-oriented memory image.
3. 3. The method as claimed in claim 1, wherein the dynamic data structure collection includes linked list type data structures.
4. 4. The method as claimed in claim 1, wherein a repeated pattern includes ancestor nodes of a same object type.
5. 5. The method as claimed in claim 1, including: merging the grouped identified objects into a virtualized object.
6. 6. The method as claimed in claim 5, wherein the virtualized object is graphically represented in a display of the resultant flat data structure.
7. 7. The method as claimed in claim 5, wherein nodes in the dynamic data structure collection have values of the size of the node, and the method including: determining a value for the size of the virtualized object by summing the total sizes of the identified objects.
8. 8. The method as claimed in claim 7, wherein applying a memory leak algorithm to the resultant flat data structure applies an algorithm that determines a drop in size of nodes from a parent node to a child node.
9. 9. The method as claimed in claim 1, including: representing as child nodes of the virtualized object, the objects referenced by the identified objects.
10. 10. The method as claimed in claim 9, wherein applying a memory leak algorithm to the resultant flat data structure applies an algorithm that determines a parent node has a large number of child nodes.
11. 11. A computer program product stored on a computer readable storage medium for detecting a memory leak, comprising computer readable program code means for performing the steps of: identifying repeated patterns of a same object type in a dynamic data structure collection; grouping the identified objects in a resultant flat data structure; and applying a memory leak algorithm to the resultant flat data structure.
12. 12. A method of providing a service to a customer over a network, the service comprising: identifying repeated patterns of a same object type in a dynamic data structure collection; grouping the identified objects in a resultant flat data structure; and applying a memory leak algorithm to the resultant flat data structure.
13. 13. A system for detecting a memory leak, comprising: a processor; means for identifying repeated patterns of a same object type in a dynamic data structure collection; means for grouping the identified objects in a resultant flat data structure; and means for applying a memory leak algorithm to the resultant flat data structure.
14. 14. The system as claimed in claim 13, wherein the dynamic data structure collection is an object-oriented memory image.
15. 15. The system as claimed in claim 13, wherein the dynamic data structure collection includes linked list type data structures.
16. 16. The system as claimed in claim 13, wherein a repeated pattern includes ancestor nodes of a same object type.
17. 17. The system as claimed in claim 13, including: means for merging the grouped identified objects into a virtualized object.
18. 18. The system as claimed in claim 17, including means for graphically representing the virtualized object in a display of the resultant flat data structure.
19. 19. The system as claimed in claim 17, wherein nodes in the dynamic data structure collection have values of the size of the node, and the system including: means for determining a value for the size of the virtualized object by summing the total sizes of the identified objects.
20. 20. The system as claimed in claim 17, including: means for representing as child nodes of the virtualized object, the objects referenced by the identified objects.