WO2008067169A9

WO2008067169A9 - Method and apparatus for modifying output dna analysis results using an expert system

Info

Publication number: WO2008067169A9
Application number: PCT/US2007/084656
Authority: WO
Inventors: John Douglas Birdwell; Tse-Wei Wang; Dale V Stansberry; Jared Pendleton
Original assignee: Univ Tennessee Res Foundation; John Douglas Birdwell; Tse-Wei Wang; Dale V Stansberry; Jared Pendleton
Priority date: 2006-11-16
Filing date: 2007-11-14
Publication date: 2009-05-14
Also published as: GB2457200A; WO2008067169A3; GB0910042D0; WO2008067169A2

Abstract

Displays of a plurality of data comprising DNA profile analysis results wherein a given DNA profile analysis result is based on a decision made by an expert system according to a modifiable rule base comprising peak expert rules. At least one peak expert rule comprises a parameter having a modifiable value and the same or another peak expert rule being selectably enabled. Allele data requiring inspection are highlighted. Input is accepted from a source, wherein the input may cause the DNA profile analysis results to be modified, expanded, accepted, viewed in alternative displays or rejected. The DNA profile analysis results may be modified by re-applying the rule base or applying a modified rule base including a rule having a modified parameter value or a rule being deselected. In response to the input, an updated table is created for output or display, wherein the updated table comprises the modified DNA profile analysis results.

Description

METHOD AND APPARATUS FOR MODIFYING OUTPUT DNA ANALYSIS RESULTS USING AN EXPERT SYSTEM

[01 J This application claims priority to the following five co-assigned applications: A METHOD OF ORGANIZING AND PRESENTING DATA IN A TABLE, U. S. Serial No. 11/560,566, filed November 16, 2006; A METHOD OF ENHANCING ACCESS TO AN ITEM REQUIRING INPUT, U. S. Serial No. 11/560,576, filed November 16, 2006; A METHOD OF ENHANCING EXPERT SYSTEM DECISION MAKING, U. S. Serial No. 11/560,580, filed November 16, 2006; A METHOD OF DISPLAYING AND INTERACTING WITH A COLLECTION OF DATA, U. S. Serial No. 11/560,588, filed November 16, 2006; and A METHOD OF INTERACTION WITH AN AUTOMATED SYSTEM, U. S. Serial No. 11/560,601, filed November 16, 2006. The contents of these five applications are herein incorporated by reference in their entirety.

[02] The U.S. government retains certain rights to this invention due to funding provided by contract J-FBI-03-196 awarded by the Department of Justice, Federal Bureau of Investigation.

TECHNICAL FIELD OF THE INVENTION

[03] The technical field of the present invention relates to the interpretation of data by an automated system and provides methods of enhanced interaction with an automated system.

BACKGROUND

[04] Automated systems are known for data analysis and display of data in numerous fields of interest. For example, in forensic science it is common for automated

1

SGRDC\3O5386 I

RECTIFIED SHEET (RULE 91) ISA/EP accurate tools to assist in the identification of biological specimens when using automated DNA profiling technology.

[05] A forensic scientist may obtain a DNA profile from a sample obtained from a personal effect of a missing person such as a toothbrush, razor, or comb, and searches for a match in a database containing DNA profiles from unknown biological specimens of a missing person or victim's remains. One of many methods of obtaining a DNA profile are described below. One can extract DNA from an unknown biological specimen by using any DNA extraction technique. Many techniques for extracting DNA are well known in the art. See, e.g., Gurvitz et al. Australas Biotechnol. 1994 Mar-Apr;4(2):88-91; Ma et al. J Forensic Sci Soc. 1994 Oct-Dec;34(4):231-5; Laber et al. J Forensic Sci. 1992 Mar;37(2):404-24. After DNA extraction, an amplification procedure such as the polymerase chain reaction can amplify the DNA with primers specific for various regions of interest. Most commonly, the regions of interest correspond to the polymorphic short tandem repeat (STR) loci of chromosomal DNA, which include D3S1358, vWA, FGA, D8S1179, D21S11, D18S51, D5S818, D13S317, D7S820, D16S539, THOl, TPOX, CSFlPO, Penta D, Penta E, D19S433, and AMEL. The amplification procedure may occur with fluorescently modified nucleotides, creating amplified DNA that is fluorescent. The fluorescent DNA is then separated by electrophoresis and the size of the DNA amplification product is determined subsequently by applicable software, allowing identification of the STR loci.

[06] WO 99/53423 published 21 October 1999 discloses "An Expert System for Analysis of DNA Sequencing Electropherograms," of Miller and Karger. The expert system can be used for real-time base-calling, or applied offline after data acquisition is complete. Meta-rules are described for determining when a rule should be permitted to fire and a set of base-calling rules are described.

[07] There is a need in the art, for example, for automated DNA STR analysis and methods for reducing workload on forensic scientists. A high workload, as is often the case after a terrorist incident, or a natural disaster, requires an extremely efficient workspace. Proper organization and presentation of data is crucial to ensure proper interpretation of results. Thus, technologies are needed in forensic science in addition to other fields that can enhance interaction with an automated system, and provide more efficient methods of organization and presentation of data.

SUMMARY OF INVENTION

[08] Several embodiments are discussed herein which provide various methods of interaction with an automated system. In one embodiment, an item is organized and presented by way of displaying a table, wherein the table displays a plurality of data comprising analysis results characterizing the at least one item, and wherein an analysis result is based on a decision made by an expert system according to a rule base. Input is accepted from a source, wherein the input may cause the analysis results to be modified, and wherein the results may be modified by re-applying the rule base. In response to the input, an updated table is created, wherein the updated table comprises the modified analysis results. This updated table is then displayed.

[09] In another embodiment, items requiring input are compiled into a list, wherein the list comprises at least one item that requires input and analysis results characterizing the at least one item. An analysis result is based on a decision made by the expert system according to a rule base. The list is displayable and input is accepted from a source, wherein the input might cause the analysis results to be modified. The results may be modified by re-applying the rule base. In response to the input, an updated list is created which comprises modified analysis results which characterize the at least one item.

[10] In another embodiment, a method of enhancing expert system decision making through interaction with at least one source is provided. An expert system is presented with at least one problem and the expert system makes at least one first decision that attempts to resolve the at least one problem, wherein the first decision results from applying a rule base for the at least one problem. Input is accepted from at least one source and in response to the input the rule base might be re-applied. A second decision that attempts to resolve the at least one problem might result from reapplying the rule base. The second decision is different from the first decision. [11] In another embodiment, a method of displaying and interacting with a collection of data is provided. The data comprise analysis results for at least one item. A first view displays a portion of the collection of data, wherein the first view comprises an expandable tree, the expandable tree being displayable in a vertical frame within a project window. A table characterizing the at least one item is displayable in tabular form in a main frame adjacent to a vertical frame, wherein the table displays the analysis results characterizing the at least one item in response to selection within the expandable tree. The analysis results of the table in this view are modifiable by input from a source to create an updated table, wherein the updated table comprises the modified analysis results. A second view of a portion of a collection of data is displayed, wherein the second view comprises a table, wherein the table is organized in tabular form with a vertical column representing the at least one item. The table displays a plurality of data comprising analysis results characterizing the at least one item, wherein the analysis results of the table are modifiable by input from a source to create an updated table, wherein the updated table comprises the modified analysis results. The updated table is then displayable. A third view of a portion of the collection of data comprises a list, wherein the list comprises at least one item requiring input from a source and analysis results of the at least one item. Analysis results are modifiable by the input to create an updated list, wherein the updated list comprises the modified analysis results. The updated list is then displayable.

[12] In an additional embodiment, a method of analyzing DNA electrophoresis data is provided. An expert system receives input from at least a first source. Data is imported and analyzed by an expert system, wherein the expert system makes at least one first decision, which characterizes the data based on a rule base. The at least one first decision is displayable and modifiable by a first input from a first source. In response to the first input from the first source, the rule base may be re-applied to make at least one second decision, wherein the at least one second decision is different from the at least one first decision, or the at least one first decision may be accepted. The at least one first decision or the at least one second decision is then displayable and modifiable in response to a first input from a second source. In response to the first input from the second source, the rule base is either re-applied to make at least one third decision, wherein the third decision is different from the second decision, or either the first or second decisions are accepted. [13] Other embodiments of the invention are computer-readable media, which store computer executable instructions for performing any of the disclosed methods.

BRIEF DESCRIPTION OF THE DRAWINGS

[14] The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings.

[15] FIG. 1 is a window illustrating an organized table comprising data.

[16] FIG. 2 is a window showing an expandable item.

[17] FIG. 3 is a window showing an organized table with options for expanding an item and providing input.

[18] FIG. 4 is a project browser window showing an expandable tree, an organized table, and a data graph.

[19] FIG. 5 is a window illustrating an organized table comprising data that has been highlighted or marked for inspection.

[20] FIG. 6 is a window illustrating an organized list comprising data that requires inspection. [21] FIG. 7 is a window showing an organized list with options for expanding an item and providing input.

[22] FIG. 7.1 is a window displaying options for modifying customization parameters for an expert system.

[23] FIG. 7.2 is a window displaying options for modifying customization parameters for an expert system.

[24] FIG. 7.3 is a window displaying options for modifying customization parameters for an expert system. (18) FIG. 4MAP is a map showing the components of a project browser window showing an expandable tree (FIG. 4A), an organized table (FIG. 4C), and a data graph (FIG. - 4B).

(20] FIG. 6 is a window illustrating an organized list comprising data that requires inspection. (21] FIG. 7 is a window showing an organized list with options for expanding an item and providing input.

(22] FIG. 7.1 is a window displaying options for modifying customization parameters for an expert system.

(23] FIG. 7.2 is a window displaying options for modifying customization parameters for an expert system.

(24] FIG. 7.3 is a window displaying options for modifying customization parameters for an expert system.

[25] FIG. 7.4 is a window displaying options for modifying customization parameters for an expert system.

[26] FIG. 7.5 is an example graph displaying a potential "stutter peak."

[27] FIG. 8 is an example computer screen comprising one of many possible configurations of simultaneously displaying a first (FIG. 8A), second (FIG. 8C), and third view (FIG. 8B) into a collection of data.

(28] FIG. 9 is a flow diagram representing an example project flow of one embodiment.

[29] FIG. 10 illustrates a block diagram of a hardware environment that may be used according to an illustrative embodiment of the invention.

RECTIFIED SHEET (RULE 91) ISA/EP analyze raw data when analyzing a time series because the assignments made for each peak are determined by peak attributes that can only be obtained from the raw data. Analyzing raw DNA peak data is described in more detail in PCT application US2006/029434 filed 28 July 2006 which claims priority to U.S. Provisional Application 60/709,424, filed 19 August 2005, both of which are incorporated herein in their entirety.

[32] It is preferred that an expert, either a human expert or an expert system, be utilized to categorize and interpret DNA data. Experts use shape and other peak features, including the relationships between peaks, to make decisions about whether peaks correspond to DNA alleles, or are artifacts due to the amplification process such as bleed-through, dye blobs, spikes, saturation, or other effects. In the following embodiments, the term "decision" is meant to comprise any decision or observation made on a peak, locus, profile, or specimen. An expert may also provide analysis for factors such as, but not limited to, contamination, procedural errors, temperature and electrical effects, and effects introduced by reagents and capillary aging.

[33] The following discussion describes aspects in which an expert can be assisted by automated means, for example, by providing an organized table for displaying data, an organized list for displaying items that require attention, a method of updating various views and displaying items in a more efficient way. Also, methods are provided where external sources can provide input which may cause an automated system to modify its decisions via a modifiable rule base.

[34] In one embodiment, the term "item" as defined by the present invention pertains to any projects, specimen, data, such as Profile: Kl (P) Inj 1, or any piece of information which might be organized. A plurality of items may be analyzed when working with data such as DNA data and it is imperative to maintain a workstation with excellent organization. For example, Figure 1 shows an item 1, which comprises specimen Kl, wherein DNA from the specimen has been extracted, amplified via polymerase chain reaction, and separated via capillary electrophoresis. Figure 1 displays a table 5 that is organized to present at least one item 1 to a source. A "source," as referred to by the present embodiment, includes but is not limited to an examiner, an expert, an expert system, a technician or other laboratory personnel, or a reviewer. In one embodiment, an examiner will make any initial decisions and a reviewer will review the examiner's work and provide feedback. Analysis results 10 characterize the item 1, and are displayable within a table 5. Characterizing an item is defined as providing information about an item. In this example, analysis results include but are not limited to allele data, such as: locus information for D3S1358, vWA, FGA, Amelogenin, D8S1179, D21S11, D18S51, D5S818, D13S317, D7S820, D16S539, THOl, TPOX, and CSFlPO. In addition, information relating to the status of the item, information pertaining to the item and information pertaining to processing of an item are also considered analysis results. One aspect of the embodiment is to display the analysis results of an item in the table 5 within a vertical column 15.

[35] The item 1 may be expanded to display source or supporting data. Expanding an item, as taught by the present embodiment, comprises displaying at least a second vertical column adjacent to the first vertical column representing an item 1. Figure 3 shows the second vertical column 30 may display source or supporting data. Source data comprises data which gives rise to the analysis results. Supporting data might include the source data as well as intermediate data that gives rise to the final analysis results. In another embodiment, source data might refer to the actual origin of data. Expanding an item can be accomplished by clicking on an item 1 or an analysis result 10 and selecting "Show Sources" 25. Figure 2 shows clicking 20 on an item 1 to select "Show Sources" 25. Figure 3 shows that item 1 has been expanded to show KitP and KitC 35 in second vertical column 30. Figure 3 also shows selection of "17 peak #6," which allows a user to, for example, "Explore in Browser" 45, "View Observations," "Reject/flag Item" 50, "view audit trail," or "edit call" 55. Source data can be further explored, or expanded in a project browser. Figure 4 shows a "project browser" 60, or the first view, which is characterized by an expandable tree 65 comprising at least one item, wherein the expandable tree is displayable in a vertical frame 70 within a project window 75. The project browser comprises a table 80 displayed in a main frame 82 adjacent to the vertical frame 70. The main frame might comprise a plurality of data and information including but not limited to a table 80 or data such as a graph 84 representing a DNA peak which was analyzed by a DNA peak analysis program. THOl, TPOX, and CSFlPO. In addition, information relating to the status of the item, information pertaining to the item and information pertaining to processing of an item are also considered analysis results. One aspect of the embodiment is to display the analysis results of an item in the table 5 within a vertical column 15.

[35] The item 1 may be expanded to display source or supporting data. Expanding an item, as taught by the present embodiment, comprises displaying at least a second vertical column adjacent to the first vertical column representing an item 1. Figure 3 shows the second vertical column 30 may display source or supporting data. Source data comprises data which gives rise to the analysis results. Supporting data might include the source data as well as intermediate data that gives rise to the final analysis results. In another embodiment, source data might refer to the actual origin of data. Expanding an item can be accomplished by clicking on an item 1 or an analysis result 10 and selecting "Show Sources" 25. Figure 2 shows clicking 20 on an item 1 to select "Show Sources" 25. Figure 3 shows that item 1 has been expanded to show KitP and KitC 35 in second vertical column 30. Figure 3 also shows selection of "17 peak #6," which allows a user to, for example, "Explore in Browser" 45, "View Observations," "Reject/flag Item" 50, "view audit trail," or "edit call" 55. Source data can be further explored, or expanded in a project browser. Figure 4MAP shows a map showing the components of a window "project browser" 60, or the first view, which is characterized by an expandable tree 65 (FIG. 4A) comprising at least one item, wherein the expandable tree is displayable in a vertical frame 70 within a project window 75. The project browser comprises a table 80 (FIG. 4C) displayed in a main frame 82 adjacent to the vertical frame 70. The main frame might comprise a plurality of data and information including but not limited to a table 80 (FIG. 4C) or data such as a graph 84 (FIG. 4B) representing a DNA peak which was analyzed by a DNA peak analysis program.

[36) An additional feature of the current embodiment is that a system, such as an expert system, which is involved in DNA peak analysis, might be required to make a decision. To enhance this process, the system may request input from a source. This

RECTIFIED SHEET (RULE 91) ISA/ EP launched or displayed at any time during the use of the program. A window is defined as a rectangular viewing area on a screen.

[38] This embodiment accepts input from a source, which has the ability to cause a system, such as an expert system, to re-apply its rule base, or selected portions of its rule base, or a rule base including changed parameter values possibly coming to different conclusions, decisions, or observations from the initial conclusions, decisions, or observations made by the system. Input may cause an analysis result to change and therefore this change needs to be incorporated into a table 5. An updated table is created when analysis results change and the updated table is then displayable to the source. The term displaying, as used by the present invention, might involve displaying information or analysis results to a user. It is also possible for information to be displayed to another computer system, such as an expert system via a communications interface 522 (Fig. 10).

[39] Figure 6 shows an additional embodiment that involves a method of enhancing access to at least one item, or a plurality of items that may require input from a source. As described above, an expert may require additional expertise for certain problems, and therefore may require input to assist in resolving any problems. To enhance access to items requiring input, a list 115 is compiled which comprises at least one item that requires input. The list may also display analysis results, portions of analysis results, or observations 135 pertaining to analysis results. The list may be displayed in a window 120. Additionally, the user inspection status can be displayed within the window. User inspection status 125 displays information pertaining to the progress of a source in reviewing, editing or providing feedback for items in the list. Comments from a source may be displayable in a text field 130. Comments may include and are not limited to the source input, or a description of the source input. The list also contains a state field 132 which displays the current state of an item, e.g., Accept, Inspect, or Reject. A source field 134 is also displayable. The source field comprises information relating to the source of the observation e.g., Examiner, Reviewer, or an expert system (e.g. STRESP). The source field displays the origin of an observation made on an item. [40] Figure 7 shows that the at least one item of the list may be expanded or explored to display source and supporting data, wherein the source and supporting data further characterize the item. A source can expand analysis results that describe an item by selecting "explore in browser," 150 as described in other embodiments. Analysis results are displayable from the table within a separate window, such as the project browser 60.

[41] The list accepts input as detailed in other embodiments, comprising but not limited to, accepting an item 140, rejecting an item 155, or defining the item as a mixture 145. A source selects an item in the list and is able to provide input, which might cause the analysis results to be modified. The term "selects" refers to a selection made with an input device such as a mouse, trackball, or stylus, as described below. In response to accepting input, an updated list is created. The updated list comprises the modified analysis results characterizing the items that required user input. The updated list is then displayable.

[42] As described above, the expert system can be presented with at least one problem wherein the expert system attempts to resolve the at least one problem. Decisions can be made by the expert system based on a modifiable rule base for the at least one problem. The expert system might be applied to all detected fluorescent DNA peaks in at least one experiment of a project. For example, in a C++ implementation, all information related to the at least one experiment can be associated with an object in an object oriented software environment, such as a Run object. In a similar manner, all project information can be associated with the object such as a project object. This application of the expert system creates observations associated with the peaks, which may be stored as C++ Observation objects in a C++ Peak object. The expert system is implemented using rules. Each rule can be thought of as an IF ... THEN clause, meaning that there is a set of conditions that are tested and must be satisfied (the IF portion), in which case actions are taken (specified in the THEN portion). Each rule has subsequent requirements that are evaluated during the validation process to determine the eventual disposition of a rule's actions. For example, the Stutter Rule, if enabled, (as further discussed below) checks to see if the primary peak is a callable allele after all dependency checks have been evaluated. In one embodiment, these actions can create one or more C++ Observations object and associate these objects with other objects such as a peak.

[43] As an example, the expert system's rules are implemented by the C++ runRule ( ) member functions that are associated with each type of observation (class derived from the C++ class Observation). These rules can be grouped into the type of object with which they are associated: specimens, profiles, loci, runs, trace data, and peaks. At various times, according to the stage of execution of the expert system and the type of objects, the conditions associated with these rules are tested for each of their corresponding observations, and the rules are fired (executed) when the conditions match.

[44] This approach is different from that employed by most rule-based expert systems. Usually, a general-purpose execution engine is implemented often using the Rete algorithm (Charles Forgy. Rete: A fast algorithm for the many pattern / many object pattern match problem. Artificial Intelligence, 19:17-37, 1982.) and operates upon a separately defined rule base of if-then rules. The constraints of the DNA profile peak fitting application require a different approach, because a tight coupling between functions best implemented in a procedural language (C++) and by a traditional rule base was necessary.

[45] The expert system's rules depend upon, and their operation can be tuned by adjusting a set of parameters that are maintained in a database with each project. The parameters are specified by a C++ AnalysisParams object. These parameters can be easily customized to the needs of each laboratory or site. The expert system's decision processes will change as a function of the selected parameters, and selected values must be carefully chosen using an informed process. Validation of the expert systems' operation using tests with each laboratory's or site's data must also be conducted to ensure that its operation is consistent with the rules defined for that lab or site. The parameters may be modified or selectably enabled for the site or for individual projects, by the person with Administrator privileges using the procedure documented in the Administrator's manual. [46] Figure 7.1 and 7.2 show that a source can change parameters to customize the expert system rule base by accessing analysis properties. For example, a source can change the parameters for "background noise" 156 by setting a "Noise peak amplitude threshold (RFU)" 157, "Noise peak normalized MSE threshold" 158, "Noise peak area threshold (RFU)" 159, and "Noise peak skew threshold (absolute val)" 160. As another example, stutter peak parameters 161 can be changed by setting a "Saturated stutter peak height threshold (% of primary)" 162. These parameters will determine how the modifiable rule base of the expert system is applied.

[47] Further, Figure 7.3 and 7.4 show that the interactive capabilities of the expert system are further modifiable by accessing application properties. For example, parameters such as "remember window layout" 163, "auto save project after data import" 164, "auto save project after analysis" 165, and "auto save interval (minutes)" 166 may be modified according to preference. As discussed above, highlighting may be utilized to draw attention to an item. The parameters that determine which colors are used for highlighting may be altered via application properties. For example, the background color of an item marked for inspection may be set to orange 167, and the text color of an item marked for inspection may be set to black 168.

[48] In a C++ implementation, the expert system's rules are executed from the C++ Specimen, Profile, Run, TraceData, Peak, and Locus classes, according to the type of object being evaluated. When a rule fires (executes), an observation is created and associated with the evaluated object. These observations are usually marked as NOT V ALIDATED; each observation indicates that characteristics of the object indicate a possible conclusion, for example that a peak is a stutter peak, but this must be verified by further automated analysis, and in certain circumstances, user review. After all initial observations are made, the validate( ) methods of the various observations are called to determine whether each observation is VALID NOT V ALID, or AMBIGUOUS (requiring human review). This reconciliation process analyzes all of the observations of a given peak for dependencies on other peaks. All dependencies to an observation associated with a peak are resolved before the peak (and its observations) are validated (marked VALID or NOT V ALID). Validation processes occur throughout the expert system's execution as required by both the initial observations and dependencies among objects. For example, all observations that are initially marked NOT V ALIDATED are evaluated using the validate( ) function for that observation type to attempt a disposition. However, peaks may depend upon the status of other peaks, so invalidation (NOT V ALID status) of a peak may cause invalidation of other peaks. Thus, as the status of objects are changed, other objects must be checked. While most of this discussion pertained to peaks, these processes are equally applicable to other objects and to their classes when implemented in an object-oriented programming language.

[49] The expert system maintains information on dependencies and automatically attempts validation wherever necessary. Once all the dependencies and observations are validated for each peak (or other object), the peak is assigned a type that is associated with the valid observations. (Peaks may have more than one type.) Circular dependencies are possible and are automatically identified by the expert system. When a circular dependency is detected and can not be automatically resolved, a flag is raised that causes notification to the source that manual review, or input, is required. This may be considered marking an item for inspection. Objects at any level of the hierarchy can have a ValidationState, and this state can be VALID only when all objects at lower levels descending from the object have been reconciled. (Note that it is not necessary for all objects to have a VALID state; for example, a peak observation with a NOT-VALID state simply means that observation is not valid and thus does not propagate.) The hierarchy does, however, allow the expert system software to attempt validation, or reconciliation, of all objects below a given object in the hierarchy by requesting the validation of that object (and implicitly of all objects below it).

[50] The expert system's rules are grouped according to function. The software design of the DNA peak calling expert system can allow observations to be associated with any object in the system, such as Specimen, Profile, Run, TraceData, Peak, and Locus objects. Within each group, rules are applied to every object of that type.

[51] In this embodiment, by way of example only, a rule base may be applied to classify "stutter peaks." The Stutter Rule, if enabled, examines the relationship between two peaks to determine if one of the peaks is a stutter peak of the other. In this discussion, the peaks will be referred to as Peak 1 and Peak 2. The Stutter Rule is only checked for peak pairs where the location of Peak 1 , in base-pairs, is less than the location of Peak 2. At least one of the peaks representative of an allele must be called, and neither peak can be saturated, in order for a stutter observation to be made.

[52] Peak 1 may be classified as a stutter peak only if the difference in the locations of the two peaks is within a specified range (in base-pairs), and if the height of Peak 1 is no more than a specified percentage of the height of Peak 2. This rule defines a region, relative to the main peak (Peak 2) in which Peak 1 must be (e.g., its location and height) in order for a stutter observation to be made. Fig. 7.5 illustrates this relationship between the peaks. The maximum height of the stutter peak is specified as a percentage of the height of the primary peak, Peak 2. The lower and upper bounds on the stutter peak's location, in base-pairs, are specified as an (negative) offset (to the left in the illustration) from the location of the main peak, Peak 2.

[53] The specified range and percentage depend upon the locus of the peaks, so the Stutter Rule first determines this locus. If Peak 1 is a called peak (has a locus and allele assignment), its locus assignment is used; otherwise, the locus assignment of Peak 2 is used. (One of the peaks must be called in order for the Stutter Rule to be executed.)

[54] The difference in the peaks' locations (calculated as the location of Peak 1 less the location of Peak 2 and therefore a negative value) and the peaks' heights are then used in a conditional expression to determine if the rule is to fire and create a stutter observation (a StutterObservation object in the example C++ implementation). The following three inequalities must all be satisfied:

clif f >-■^■■ m_s tutlow ά i £ £ -• ; ^■-^■-^■-^■- sa~s tutkigh [Peak 1

where diff is the (negative) difference in the peaks' locations, m stutlow is the specified smallest allowed (negative) difference in locations between the stutter peak and its primary peak, m_stuthigh is the specified largest allowed (negative) difference in locations between the stutter peak and its primary peak, and m stutratio is the maximum specified ratio between the height of a stutter peak and the height of its associated primary peak. These three parameters are specified independently for each locus. The default values of these parameters are determined, first, by the initialization of the database, which occurs in the example C++ implementation of theStrEspApp class in the dblnsertLocusRef ( ) member function, and second by any actions a user (with appropriate privilege) may take to change default values for an analysis kit.

[55] The expert system described in this embodiment has the capability to notify a source to inspect, or enter input, to assist in the expert system decision making process. Based on various observations made by the expert system, the expert system may require additional expertise from a source for at least one problem. Input from a source can trigger the expert system to re-apply the rule base for the at least one problem. By way of example, the at least one problem is whether the DNA peak is a stutter peak or not. Initially, the system, or expert system may make a first decision which classifies a peak as a stutter peak; however, if pre-determined criteria are not met, the system may request user input for assistance in the decision making. If a source disagrees with the system, the source may provide input, which causes the rule base to be re-applied to make a second decision. The second decision might then be different from the first decision.

[56] Rather than re-apply the entire rule base, as applied to the DNA data, one aspect of this embodiment is to only re-apply those portions of the rule base for the at least one problem that would result in a second decision to resolve the at least one problem. For example, if a source rejects an expert system made observation of "stutter peak," the expert system will re-apply only the rule base as applied to stutter peaks, or potential stutter peaks and to objects that depend upon the disposition or decisions involving the stutter peak. It might not re-apply other rule bases for different problems such as noise rule, broad peak rule, spike rule, duplicate allele rule, etc. One method in which this selective re-application of the rule base can be achieved is by analysis of the dependencies among objects. These dependencies, for example, can be represented by a graph, which can be analyzed to determine which rule base needs to be re-applied. The advantage of selective re-application is performance; the software responds much more rapidly to user, or source, actions or input.

[57] Figure 8 refers to an additional embodiment which involves a method of displaying and interacting with a collection of data, wherein the data comprise analysis results for the at least one item. A collection of data refers to the data that is being utilized in an analysis. This may include an entire data set, or a fraction thereof. This embodiment includes features of previously described embodiments. It is preferred to have at least one view, or a plurality of views, available to a source, wherein each view may contain the data organized in different ways. This can enhance efficiency and organization by allowing a source to arrange the plurality of views according to preference. For example, a first view (169 - project browser) comprises an expandable tree 170 comprising at least one item. The expandable tree is displayable in a vertical frame 175 within a project window. If an item requires input, as potentially determined by an expert system (described above), the item may be highlighted within a table 180. A table within the first view comprises analysis results characterizing the at least one item and are displayable in a main frame 185 adjacent to the vertical frame 175. The table within the first view 190 displays the analysis results characterizing the at least one item responsive to selection within the expandable tree 170. The analysis results displayed in the table 190 are modifiable by input from a source to create an updated table within the first view, wherein the updated table comprises the modified analysis results. The computer system (detailed below) may then display the updated table to the user in the first view 169.

[58] A second view 195 of a portion of the collection of data comprises a table 225 which is organized in tabular form with a vertical column 200 representing the at least one item. The table within the second view 225 displays a plurality of data comprising analysis results 210 characterizing the at least one item, wherein the analysis results of the table are modifiable by input to create an updated table, which comprises the modified analysis results. The updated table is then displayable in the second view 195.

[59] A third view of a portion of the collection of data comprises a list 215. The list contains at least one item that requires user input in addition to analysis results of the stutter peak or not. Initially, the system, or expert system may make a first decision which classifies a peak as a stutter peak; however, if pre-determined criteria are not met, the system may request user input for assistance in the decision making. If a source disagrees with the system, the source may provide input, which causes the rule base to be re-applied to make a second decision. The second decision might then be different from the first decision.

(57) Figures 8A-8C refer to an additional embodiment which involves a method of displaying and interacting with a collection of data, wherein the data comprise analysis results for the at least one item. A collection of data refers to the data that is being utilized in an analysis. This may include an entire data set, or a fraction thereof. This embodiment includes features of previously described embodiments. It is preferred to have at least one view, or a plurality of views, available to a source, wherein each view may contain the data organized in different ways. This can enhance efficiency and organization by allowing a source to arrange the plurality of views according to preference. For example, a first view (FIG. 8A) (169 — project browser) comprises an expandable tree 170 comprising at least one item. The

18

RECTIFIED SHEET (RULE 91) ISA/ EP expandable tree is displayable in a vertical frame 175 within a project window. If an item requires input, as potentially determined by an expert system (described above), the item may be highlighted within a table 180. A table within the first view comprises analysis results characterizing the at least one item and are displayable in a main frame 185 adjacent to the vertical frame 175. The table within the first view 190 displays the analysis results characterizing the at least one item responsive to selection within the expandable tree 170. The analysis results displayed in the table 190 are modifiable by input from a source to create an updated table within the first view, wherein the updated table comprises the modified analysis results. The computer system (detailed below) may then display the updated table to the user in the first view 169.

[58] A second view 195 (FIG. 8C) of a portion of the collection of data comprises a table 225 which is organized in tabular form with a vertical column 200 representing the at least one item. The table within the second view 225 displays a plurality of data comprising analysis results 210 characterizing the at least one item, wherein the analysis results of the table are modifiable by input to create an updated table, which comprises the modified analysis results. The updated table is then displayable in the second view 195.

[59] A third view (FIG. 8B) of a portion of the collection of data comprises a list 215. The list contains at least one item that requires user input in addition to analysis results of the at least one item. As described in the above embodiments, the analysis results are modifiable by input to create an updated list. The updated list comprises the modified analysis results which may characterize the at least one item. The updated list is displayable in the third view 220.

[60] It is advantageous to display at least two of the first, second, and third views simultaneously in any possible configuration on a computer screen. Each view may be displayed in a window and each view's location on a screen can be modified by a source. Modifying a view's location on a screen might be done through moving the

19

RECTIFIED SHEET (RULE 91) ISA/ EP first decision. The second decision is also modifiable as a result of input. A second source optionally logs into the program and reviews the first and/or second decision. A second source can then enter a first input. As described above, possible inputs include, without limitation, acceptance or rejection of the first or second decision indicating agreement or disagreement, respectively, with the input of the first source or the first or second decision. Input from the second source can also provide feedback to the first source or expert system.

[65] In one example, the system may be configured to allow the second source to enter a first input comprising feedback, which might be agreement or disagreement with the decision provided by the expert system and/or first source. In this configuration, only inputs provided by the first source can result in direct or indirect modification of a decision. It is preferable that the expert system utilize input from the second source to determine whether the first and second source are in agreement with each other. In this instance, the system can be configured to enhance the workflow process involving the two sources and system, and can ensure that consensus between the two sources and the system be obtained. For example, agreement can be indicated by input from the first source and agreement with the re-analysis performed by the expert system.

[66] In one embodiment, the first input from the second source might indicate disagreement with the first source and cause the expert system to re-evaluate the rule base and make at least one third decision by modifying the second decision. The third decision might be a direct result of the first input from the second source if the system is configured to allow this behavior. However, in another embodiment, the third decision could be an indirect result of the first input from the second source. In this embodiment, the first source examines the at least one first and/or second decisions, first input from the first source and/or the feedback and first input from the second source, and provides a second input to the system. The expert system then makes at least one third decision by modifying the second decision. Thus, feedback from the second source can indirectly modify the second decision. During this process, the first and second source may optionally choose to examine other information by, for example, examining data maintained by the system that is related to the decisions and/or inputs. The objective of this process is for the first and second sources to reach agreement and for a decision to be made that is consistent with this agreement.

[67] Alternatively, the second source might agree with the first source review and the corresponding first or second decision by, for example, accepting the decision or providing an indication of concurrence in feedback. In any scenario, once consensus has been reached and there is agreement on a decision, the project can be finalized.

[68] In another embodiment, the expert system may be configured to interact with one source. As described above, the source might include an examiner, a reviewer, or another computer system, such as an expert system. In this scenario, the one source has the ability to modify the at least one decision made by the expert system. The source has the ability to cause the expert system to re-evaluate, or re-apply the rule base to make at least one second decision or the source may accept the first decision and finalize the project.

Example Project Flow

[69] One example of possible project flow, as shown in Figure 9, is discussed below. The user who creates a project and performs the initial analysis and results inspection will be referred to as the examiner and the user who opens the inspected project for review will be referred to as the reviewer. After starting the program, the examiner optionally logs in 250 to have his or her personal privileges and preferences automatically loaded by the DNA profile peak (allele) calling expert system. A new project can then be created 260 with a specified name, workgroup, and optionally, a preferred reviewer. Raw capillary electrophoresis DNA files can be imported into the project 270. The examiner may select, for each file, several properties (such as run type, kit, specimen, extraction) if there is no consistent naming convention or information available from other software, such as laboratory information management system (LIMS) otherwise, this information may be obtained from a file name convention or another source such as a LIMS or database. The project is then ready to be analyzed 280. After an automated analysis by the expert system, the examiner resolves items 290 flagged for inspection in the examiner's TDL (To Do List), also referred to as the third view, or the list, by accepting or rejecting the listed items. Observations made by the expert system may also be overridden by the examiner by editing items in the project, but the edited observation will be added to the TDL, also referred to as the third view or the list, for review. Once all items in the TDL have been inspected and resolved, the examiner notifies the reviewer 300 that the project is ready for review. The reviewer then optionally logs in to the expert system program and opens the project as a "reviewer." This implies that the reviewer will review the actions taken by the examiner. The reviewer inspects the reviewer's TDL, also referred to as the third view or the list 320, which now lists the actions taken by the examiner on each item, and "accepts," or provides "feedback" for each examiner decision. The reviewer may also "reject" the examiner's decisions. The reviewer and examiner are to discuss the "feedback" items and form a consensus on the appropriate actions to take 330. Once both the examiner and reviewer agree on actions taken on all of the items listed in the TDL, also referred to as the third view or the list, items the project can be finalized by either user 340. A project is finalized when the sources have accepted a decision. After fmalization, projects cannot be modified, but a project can later be de-finalized by either the reviewer or the examiner. Once it is finalized, reports can be generated and profiles exported in a CMF format, importable to either CODIS or POPSTAT, or as a CODIS table for import to CODIS or a compatible LIMS environment 350. In one embodiment, a first source is an examiner and a second source is a reviewer.

IMPLEMENTATION MECHANISMS - HARDWARE OVERVIEW

[70] Methods of the first embodiment and subsequent embodiments may be utilized in connection with computer readable media, which may be provided for temporary or permanent storage in a personal computer or other computer known in the art. FIG. 10 is a block diagram that illustrates a computer system 500 upon which at least one embodiment of the invention may be implemented. Computer system 500 includes a bus 502 or other communication mechanism for communicating information, and at least one processor 504 coupled with bus 502 for processing information. Computer system 500 also includes a main memory 506, such as a random access memory ("RAM") or other dynamic storage device, coupled to bus 502 for storing information and instructions to be executed by processor 504. Main memory 506 also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 504. Computer system 500 may further include a read only memory ("ROM") 508 or other static storage device coupled to bus 502 for storing static information and instructions for processor 504. A storage device 510, such as a magnetic disk, optical disk, solid-state memory, or the like, may be provided and coupled to bus 502 for storing information and instructions.

[71] Computer system 500 may optionally be coupled via bus 502 to a display 512, such as a cathode ray tube ("CRT"), liquid crystal display ("LCD"), plasma display, television, or the like, for displaying information to a computer user. Alternatively, displayable information may be delivered to a computer user or another computer system or computer program using a communication interface 518. Delivery of information to another computer is also referred to as displaying said information. An input device 514, including alphanumeric and other keys, may be coupled to bus 502 for communicating information and command selections to processor 504. An optional type of user input device is cursor control 516, such as a mouse, trackball, stylus, or cursor direction keys for communicating direction information and command selections to processor 504 and for controlling cursor movement on display 512. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane. Alternatively, information and command selections may be communicated to processor 504 using a communication interface 518. Optionally, separate communication interfaces may be used to deliver information to a computer user or another computer system or computer program, and to communicate information and command selections to processor 504.

[72] The invention is related to the use of computer system 500 for interpreting raw DNA analysis data by an automated system and providing methods of enhanced interaction with an automated system. According to one embodiment of the invention, interpretation of raw data is provided by computer system 500 in response to processor 504 executing one or more sequences of one or more instructions contained in main memory 506. Such instructions may be read into main memory 506 from another computer-readable medium, such as storage device 510. Execution of the sequences of instructions contained in main memory 506 causes processor 504 to perform the process steps described herein. In alternative embodiments, apparatus such as hard- wired circuitry may be used in place of or in combination with software instructions to implement the methods of the invention. For example, a field- programmable gate array (FPGA) or application-specific integrated circuit (ASIC) may be used. Such a device can, for example, implement associative memory to aid in indexing, search, and retrieval of information stored in a database. Thus, embodiments of the invention are not limited to any specific combination of hardware circuitry and software.

[73] The objects described in this invention, including for example objects corresponding to the C++ classes Peak, Observation, AnalysisParams, TraceData, Locus, Specimen, Profile, and Run, are typically stored in main memory 506. Other data structures may be used in place of these objects, in either an object-oriented or other programming and software execution environment. Optionally, a database, such as MySQL, Oracle, Microsoft SQL Server, or RDB, may be used to store objects or other data structures. In this case, it is advantageous to provide software to serialize and unserialize information stored in objects or other data structure in order to convert the information into and from a form that is more suitable for storage in a database. One example of this conversion is performed in order to preserve links between objects or data structures, which in a representation stored in main memory may be a memory address or pointer, and in a database may be a symbolic reference or other method of reference using, for example, a dictionary or table of symbols. A second example of this conversion is to use a representation of numbers that is not specific to a particular processor 504.

[74] When a database is used to store objects or other data structures, regardless of representation, in addition to objects or data structures stored in main memory, it can be advantageous to separate the functions of database, expert system, and interface (or views) in to two or more parts. In this case, for example, the database or expert system can be implemented or executed on a second processor 504 or computer system 500 in addition to the computer system that implements or executes the interface. The multiple computer systems can communicate and coordinate their activities across any combination of a local area network, a wide area network, or, in a multiprocessor computer, shared memory or other communications channel. In particular, this allows multiple users to simultaneously utilize the expert system or information stored in the database. For example, a client/server configuration with a single database is feasible, which allows a user to access his or her projects from any client computer system. In this case, standard and well-known methods can be utilized to control and coordinate access and enforce consistency across the server and all clients. Such a client/server configuration can utilize user interface technologies such as those discussed elsewhere in this application, or web-enabled interface technologies such as the hypertext markup language (HTML), JavaScript, Java, or a service-oriented architecture (SOA), all of which are well known in the art. In this manner, services can be provided by at least one server 530 over a large geographic area and to one or more organizations. When multiple servers are used, the servers may communicate to possibly share information and coordinate activities, including the ability to distribute workloads across multiple resources.

[75] The term "computer-readable medium" as used herein refers to any medium that participates in providing instructions to processor 504 for execution. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical or magnetic disks, solid state memories, and the like, such as storage device 510. Volatile media includes dynamic memory, such as main memory 506. Transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise bus 502. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[76] Common forms of computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other optical medium, solid-state memory, punch cards, paper tape, any other physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH- EPROM, any other memory chip or cartridge, a carrier wave as described hereinafter, or any other medium from which a computer can read. Various forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to processor 504 for execution. [77] Computer system 500 may also include a communication interface 518 coupled to bus 502. Communication interface 518 provides a two-way data communication coupling to a network link 520 that is connected to a local network 522. For example, communication interface 518 may be an integrated services digital network ("ISDN") card or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface 518 may be a network card (e.g., and Ethernet card) to provide a data communication connection to a compatible local area network ("LAN") 522 or wide area network ("WAN"), such as the Internet 528 or a private network. Wireless links may also be implemented. In any such implementation, communication interface 518 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information. For example, a forensic investigation may require a data communication connection to a database comprising at least DNA profile data or other forensic information.

[78] Network link 520 typically provides data communication through one or more networks to other data devices. For example, network link 520 may provide a connection through local network 522 to a host computer 524 or to data equipment operated by an Internet Service Provider or private network service provider ("ISP"). ISP in turn provides data communication services through a packet data communication network such as the worldwide network commonly referred to as the "Internet" 528 or a private network. An example of a private network is a secure data network linking law enforcement agencies and used for transmission of DNA and/or non-DNA information. Local network 522 and Internet 528 both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 520 and through communication interface 518, which carry the digital data to and from computer system 500, are exemplary forms of carrier waves transporting the information.

[79] Computer system 500 can send messages and receive data, including program code, through the network(s), network link 520 and communication interface 518. In the Internet example, a server 530 might transmit a requested code for an application program through Internet 528, host computer 524, local network 522 and communication interface 518. [80] The received code may be executed by processor 504 as it is received, and/or stored in storage device 510, or other tangible computer-readable medium (e.g., nonvolatile storage) for later execution. In this manner, computer system 500 may obtain application code and/or data in the form of an intangible computer-readable medium such as a carrier wave, modulated data signal, or other propagated signal.

[81] Computer system 500 can be configured using the methods of this invention to provide services across a network to forensic personnel having client computers capable of connection to the network. These services can also be provided to other software, located in either computer system 500 or a separate computer system connected by a network, network link, or communication interface to computer system 500. The services can be protected using methods of authentication and/or encryption that are known in the fields of computer science and computer security in order to ensure data are neither compromised nor disclosed and to trace all accesses to the data. The computer system 500 and other associated information storage and communication components can be protected using devices and methods that are known in the fields of computer science and computer security, such as with firewalls, physical access controls, power conditioning equipment, and backup or redundant power sources. The information stored by computer system 500 and computer-readable media can be further protected using backup or redundant information storage systems, such as those that are well-known in the art. Examples include tape storage systems and RAID storage arrays.

[82] Thus, there has been shown and described several approaches for organizing and presenting data and methods for interacting with an automated system. Approaches for re-applying an expert system rule base have also been described which might be utilized in concert with the methods for organizing and presenting data. The following set of claims should not be deemed to be limited to the embodiments described above. Alternative embodiments may come to mind to one of ordinary skill in the art for application in alternative or later generation automated systems.

[83] All patents, patent applications, and references cited in this disclosure are expressly incorporated herein by reference.

Claims

WHAT WE CLAIM IS:

1. A method of organizing and outputting at least one item, said item comprising results of a DNA analysis according to a rule base for use in an expert system, the expert system automatically outputting the at least one item according to said rule base, CHARACTERIZED BY: a. outputting a table of data, said table comprising data representing DNA analysis results based on a decision made by the expert system according to a modifiable rule base, said modifiable rule base comprising one of a rule having a modifiable parameter value and a rule having a property of being selectably enabled, the output DNA analysis results including a status of one of acceptance or inspection required and at least one data requiring inspection; b. accepting input from a source, the input causing a change of the DNA analysis results wherein the results are changed by one of re-applying said rule base or applying a modified rule base; c. creating an updated table responsive to said source input, the updated table comprising the changed DNA analysis results; and d. outputting the updated table including the changed DNA analysis results.

2. A method as recited in claim 1 FURTHER CHARACTERIZED BY organizing the table being in tabular form for display, said display having a first vertical column representing said at least one item having a label, a first entry of said at least one item comprising said status and subsequent entries comprising information defining one of concordance or concordance mixture, process and allele data.

3. A method as recited in claim 2 FURTHER CHARACTERIZED BY expanding said at least one item to display source and supporting data in a parallel vertical column to said first vertical column, wherein the source and supporting data further characterize an item.

4. A method as recited in claim 3 FURTHER CHARACTERIZED IN THAT the expansion comprises displaying at least said parallel vertical column, said parallel vertical column comprising second and third columns adjacent to the first vertical column, wherein the second and third vertical columns display first and second sources and supporting data as two vertical columns having different labels from said item label for said first vertical column.

5. A method as recited in one of claims 2, 3 or 4 FURTHER CHARACTERIZED BY highlighting said status of inspection and highlighting allele data requiring inspection.

6. A method as recited in one of claims 1, 2, 3, 4 or 5 FURTHER CHARACTERIZED IN THAT the input comprises at least one of: a. accepting the item; b. rejecting/flagging the item; and c. editing the item.

7. A method as recited in one of claims 1, 2, 3, 4, 5 or 6, FURTHER CHARACTERIZED BY receiving input from a source to define the item as a mixture and reapplying said rule base to input data.

8. A method as recited in one of claims 5, 6 or 7 FURTHER CHRACTERIZED BY at least one DNA analysis result that requires input being highlighted, receiving input from a source for selecting said highlighted result and, upon its selection, superimposing a window on said displayed table permitting a DNA allele calling result to be explored.

9. A method as recited in one of claims 5 or 7 FURTHER CHARACTERIZED IN THAT said highlighting comprises at least one of: a color change, labeling with "Inspect," labeling with a designated symbol, and labeling with parentheses.

10. A method as recited in claim 8 FURTHER CHARACTERIZED IN THAT said exploration comprises one of exploring in a browser, viewing observations, rejecting or flagging an item, viewing an audit trail and editing an allele call.

11. A method as recited in claim 8 FURTHER CHARACTERIZED BY displaying a window comprising a project browser window, said project browser window comprising an expandable tree, an organized table and a data graph.

12. A method as recited in one of claim 8 or 10 FURTHER CHARACTERIZED BY receiving input for selecting a called allele for exploration.

13. A method as recited in one of claims 1-12 FURTHER CHARACTERIZED IN THAT the DNA analysis results comprise at least four of the following: a) status of the item; b) information pertaining to the item; c) process information pertaining to the item; d) locus information for D3S1358; e) locus information for vWA; f) locus information for FGA; g) locus information for Amelogenin; h) locus information for D8S1179; i) locus information for D21 S 11 ; j) locus information for D18S51; k) locus information for D5S818; 1) locus information for D13S317; m) locus information for D7S820; n) locus information for D16S539; o) locus information for THOl; p) locus information for TPOX; and q) locus information for CSFlPO.

14. A method as recited in one of claims 1-13 FURTHER CHARACTERISED IN THAT said source comprises one of a reviewer, an examiner, or an expert system.

15. A method as recited in one of claim 2-14 FURTHER CHARACTERIZED BY displaying the table in a window among a plurality of selectable windows, one window comprising said table of at least one item, a second window including a vertical frame comprising an expandable tree displayed with one of graphically presented data and tabular presented data, and a third window comprising a list of data items requiring inspection.

16. A method as recited in claim 15 FURTHER CHARACTERIZED IN THAT said expandable tree of said vertical frame comprises two of specimens, ladders, reagent blanks, negative controls and positive controls.

17. A method as recited in claim 15 FURTHER CHARACTERIZED BY said second window including a selectable input of analysis, wherein, upon selection of said analysis, presenting an output for further selecting default analysis properties.

18. A method as recited in claim 15 FURTHER CHARACTERIZED BY receiving input for toggling among said first, second and third windows.

19. A method as recited in claim 15 FURTHER CHARACTERIZED IN THAT said third window comprises a text box for displaying comments from a source.

20. A method as recited in claim 15 FURTHER CHARACTERIZED IN THAT said third window comprises a list of action entries requiring action, observations regarding said action entries, status regarding inspection and source for action entry requiring inspection.

21. A method as recited in claim 15 FURTHER CHARACTERIZED BY updating data in one window responsive to accepting input from a source in another window.

22. A method as recited in one of claims 1-21 FURTHER CHARACTERIZED IN THAT said modifiable rule base comprises a plurality of expert rules comprising one of a rule associated with background noise, a rule for a spike peak type, a rule for a broad peak, a rule for a bleed-through peak, a rule for a pull-up peak, a rule for stutter peaks, a rule for A peaks and a rule for a global filter.

23. A method as recited in claim 22 FURTHER CHARACTERIZED IN THAT said modifiable rule base comprises a stutter peak rule, said stutter peak rule being selectably enabled and having an associated parameter value for peak height threshold.

24. A method as recited in one of claims 22 or 23 FURTHER CHARACTERIZED IN THAT said modifiable rule base comprises a background noise rule, said background noise rule being selectably enabled and having an associated parameter value for a noise threshold.

25. A method as recited in one of claims 1 -24 FURTHER CHARACTERIZED BY selectably applying properties comprising one of automatic save parameters and remembering a window layout.

26. A method as recited in one of claims 1-25, FURTHER CHARACTERIZED BY enhancing expert system decision making through interaction with at least one source, wherein said expert system is presented with at least one DNA analysis problem and wherein the expert system makes at least one first allele calling decision to resolve the at least one DNA analysis problem, the first decision resulting from applying said modifiable rule base for the at least one problem, comprising: accepting input from at least one source to obtain a second allele calling decision related to said first decision; in response to the input, re-applying the modifiable rule base for the at least one problem; and making a second allele calling decision for resolving the at least one problem, the second decision resulting from re-applying the modifiable rule base, and wherein the second decision is different from the first decision.

27. A method as recited in claim 26 FURTHER CHARACTERIZED BY said expert system re-applying those portions of said modifiable rule base for the at least one problem resulting in a second allele calling decision to resolve the at least one problem, the second decision being different from the first decision.

28. A method as recited in one of claims 1-27 FURTHER CHARACTERIZED BY selectively reapplying said modifiable rule base and analyzing dependencies among objects comprising one of a specimen, rule, locus, trace data and peak.

29. A method of organizing and outputting at least one item, said item comprising results of a DNA analysis according to a rule base for use in an expert system, the expert system automatically outputting the at least one item according to said rule base,

CHARACTERIZED BY: a. outputting a list of items, said list comprising a plurality of data representing DNA analysis results based on a decision made by the expert system according to a modifiable rule base, said modifiable rule base comprising one of a rule having modifiable parameter value and a rule having a property of being selectably enabled, the output DNA analysis results including a status of one of acceptance or inspection required and at least one item requiring inspection, the list of items comprising a list of items requiring action, observations regarding said action items, status regarding inspection and source for action item requiring inspection; b. receiving input for selecting an item of the list, and c. accepting input from a source, the input causing a change of the DNA analysis results wherein the results are changed by one of re-applying said rule base or applying a modified rule base.

30. A method as recited in claim 29 FURTHER CHARACTERISED IN THAT said source comprises one of a reviewer, an examiner, or an expert system.

31. A method as recited in one of claims 29 or 30 FURTHER CHARACTERIZED IN THAT said exploration comprises one of exploring in a browser, viewing observations, rejecting or flagging an item, viewing an audit trail and editing an allele call.

32. A method as recited in one of claims 29, 30 or 31 FURTHER CHARACTERIZED BY superimposing a window comprising selectable options of one of accepting or rejecting the selected item, exploring the item, viewing the observations and viewing an audit trail responsive to item selection.

33. A method of analyzing DNA electrophoresis data for use in a first expert system CHARACTERIZED BY the expert system receiving input from a first source and a second source where one of the first and second sources comprises a second expert system, the method comprising

(a) importing DNA electrophoresis data;

(b) analyzing the data, the first expert system making a first decision characterizing the data based on a modifiable rule base;

(c) displaying the first decision, wherein the first decision is modifiable by a first input from the first source; (d) in response to the first input from the first source, either reapplying the modifiable rule base to make a second decision, wherein the second decision is different from the first decision, or accepting the first decision;

(e) displaying the first decision or the second decision, the first or second decision being modifiable in response to an input from the second source; and

(f) either reapplying the modifiable rule base to make a third decision, wherein the third decision is different from the first and from the second decision or accepting one of the first or the second decision.

34. A method as recited in claim 33 FURTHER CHARACTERIZED BY reapplying (d) comprising reapplying selected portions of the modifiable rule base resulting in a second decision different from the first decision.

35. A method as recited in claim 33 FURTHER CHARACTERIZED BY reapplying (d) comprising modifying a parameter of a rule of the modifiable rule base resulting in a second decision different from the first decision.

36. A method as recited in one of claim 33, 34 or 35 FURTHER CHARACTERIZED BY creating an audit trail from inputs from said first and second sources.

37. A method as recited in one of claims 32-36 FURTHER CHARACTERIZED BY displaying a window comprising a project browser window, said project browser window comprising an expandable tree, an organized table and a data graph.

38. Apparatus including a processor, an input device and a memory CHARACTERIZED IN THAT said apparatus performs a method of organizing and outputting at least one item as recited in one of claims 1-28.

39. Apparatus including a processor, an input device, a memory and a display CHARACTERIZED IN THAT said apparatus performs a method of organizing and displaying at least one item as recited in one of claims 2-28.

40. Apparatus including a processor, an input device, a memory and a communications interface, said communications interface for receiving input from a source and for outputting data to one of a client or a server

CHARACTERIZED IN THAT said apparatus performs a method of organizing and outputting at least one item as recited in one of claims 1-28.

41. Apparatus including a processor, an input device and a memory CHARACTERIZED IN THAT said apparatus performs a method of organizing and outputting at least one item as recited in one of claims 29 or 30.

42. Apparatus including a processor, an input device, a memory and a display CHARACTERIZED IN THAT said apparatus performs a method of analyzing DNA electrophoresis data as recited in one of claims 33-37.

43. Apparatus including a processor, an input device, a memory and a communications interface, said communications interface for receiving input from a source and for outputting data to one of a client or a server

CHARACTERIZED IN THAT said apparatus performs a method of organizing and outputting at least one item as recited in one of claims 29 or 30.

44. Apparatus including a processor, an input device, a memory and a communications interface, said communications interface for receiving input from a source and for outputting data to one of a client or a server

CHARACTERIZED IN THAT said apparatus performs a method of analyzing DNA electrophoresis data as recited in one of claims 33-37.