US8135711B2 - Method and apparatus for sociological data analysis - Google Patents
Method and apparatus for sociological data analysis Download PDFInfo
- Publication number
- US8135711B2 US8135711B2 US12/422,902 US42290209A US8135711B2 US 8135711 B2 US8135711 B2 US 8135711B2 US 42290209 A US42290209 A US 42290209A US 8135711 B2 US8135711 B2 US 8135711B2
- Authority
- US
- United States
- Prior art keywords
- actor
- items
- actors
- item
- user
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related, expires
Links
- 238000000034 method Methods 0.000 title claims abstract description 162
- 238000007405 data analysis Methods 0.000 title description 3
- 238000004891 communication Methods 0.000 claims abstract description 142
- 238000004458 analytical method Methods 0.000 claims abstract description 38
- 230000001364 causal effect Effects 0.000 claims abstract description 5
- 230000001976 improved effect Effects 0.000 claims abstract description 4
- 230000000694 effects Effects 0.000 claims description 51
- 239000011159 matrix material Substances 0.000 claims description 16
- 230000003993 interaction Effects 0.000 claims description 9
- 238000012217 deletion Methods 0.000 claims description 7
- 230000037430 deletion Effects 0.000 claims description 7
- 230000001932 seasonal effect Effects 0.000 claims description 3
- 238000012552 review Methods 0.000 description 89
- 238000004519 manufacturing process Methods 0.000 description 67
- 238000012800 visualization Methods 0.000 description 45
- 230000008569 process Effects 0.000 description 41
- 230000000007 visual effect Effects 0.000 description 40
- 230000006399 behavior Effects 0.000 description 38
- 230000008859 change Effects 0.000 description 29
- 238000011835 investigation Methods 0.000 description 29
- 238000010586 diagram Methods 0.000 description 28
- 230000009471 action Effects 0.000 description 27
- 230000008520 organization Effects 0.000 description 26
- 230000004044 response Effects 0.000 description 17
- 238000005070 sampling Methods 0.000 description 17
- 238000004422 calculation algorithm Methods 0.000 description 15
- 230000007246 mechanism Effects 0.000 description 15
- 238000007418 data mining Methods 0.000 description 13
- 238000013459 approach Methods 0.000 description 12
- 230000004043 responsiveness Effects 0.000 description 12
- 239000003086 colorant Substances 0.000 description 11
- 238000013507 mapping Methods 0.000 description 10
- 230000036961 partial effect Effects 0.000 description 10
- 230000000699 topical effect Effects 0.000 description 10
- 230000003190 augmentative effect Effects 0.000 description 9
- 230000006870 function Effects 0.000 description 9
- 238000012986 modification Methods 0.000 description 9
- 230000004048 modification Effects 0.000 description 9
- 230000000737 periodic effect Effects 0.000 description 9
- 238000007792 addition Methods 0.000 description 8
- 230000008901 benefit Effects 0.000 description 8
- 238000003066 decision tree Methods 0.000 description 8
- 238000005192 partition Methods 0.000 description 8
- 239000000463 material Substances 0.000 description 7
- 238000011160 research Methods 0.000 description 7
- 230000007306 turnover Effects 0.000 description 7
- 238000012937 correction Methods 0.000 description 6
- 230000002596 correlated effect Effects 0.000 description 5
- 230000000875 corresponding effect Effects 0.000 description 5
- 230000002996 emotional effect Effects 0.000 description 5
- 238000007667 floating Methods 0.000 description 5
- 230000002829 reductive effect Effects 0.000 description 5
- 238000012795 verification Methods 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 4
- 230000001419 dependent effect Effects 0.000 description 4
- 238000012553 document review Methods 0.000 description 4
- 238000009877 rendering Methods 0.000 description 4
- 238000013517 stratification Methods 0.000 description 4
- 238000006467 substitution reaction Methods 0.000 description 4
- 230000008093 supporting effect Effects 0.000 description 4
- 238000013474 audit trail Methods 0.000 description 3
- 238000010276 construction Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 238000004091 panning Methods 0.000 description 3
- 238000012797 qualification Methods 0.000 description 3
- 238000005067 remediation Methods 0.000 description 3
- 230000002441 reversible effect Effects 0.000 description 3
- 238000000551 statistical hypothesis test Methods 0.000 description 3
- 238000007619 statistical method Methods 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- XOJVVFBFDXDTEG-UHFFFAOYSA-N Norphytane Natural products CC(C)CCCC(C)CCCC(C)CCCC(C)C XOJVVFBFDXDTEG-UHFFFAOYSA-N 0.000 description 2
- 230000002730 additional effect Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 2
- 238000013475 authorization Methods 0.000 description 2
- 239000003795 chemical substances by application Substances 0.000 description 2
- 230000000052 comparative effect Effects 0.000 description 2
- 239000000470 constituent Substances 0.000 description 2
- JLYFCTQDENRSOL-VIFPVBQESA-N dimethenamid-P Chemical compound COC[C@H](C)N(C(=O)CCl)C=1C(C)=CSC=1C JLYFCTQDENRSOL-VIFPVBQESA-N 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000002452 interceptive effect Effects 0.000 description 2
- 238000002372 labelling Methods 0.000 description 2
- 230000006855 networking Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000013439 planning Methods 0.000 description 2
- 230000008521 reorganization Effects 0.000 description 2
- 238000000926 separation method Methods 0.000 description 2
- 238000000638 solvent extraction Methods 0.000 description 2
- 238000001228 spectrum Methods 0.000 description 2
- 206010000117 Abnormal behaviour Diseases 0.000 description 1
- 238000012935 Averaging Methods 0.000 description 1
- 241000270666 Testudines Species 0.000 description 1
- 235000014787 Vitis vinifera Nutrition 0.000 description 1
- 240000006365 Vitis vinifera Species 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000012152 algorithmic method Methods 0.000 description 1
- 230000001174 ascending effect Effects 0.000 description 1
- 238000012550 audit Methods 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 230000004397 blinking Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000005352 clarification Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 150000001875 compounds Chemical class 0.000 description 1
- 230000001186 cumulative effect Effects 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000007123 defense Effects 0.000 description 1
- 238000000151 deposition Methods 0.000 description 1
- 230000000994 depressogenic effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000004069 differentiation Effects 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000007717 exclusion Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 239000012530 fluid Substances 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 238000013549 information retrieval technique Methods 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 238000012804 iterative process Methods 0.000 description 1
- 230000000670 limiting effect Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 239000003550 marker Substances 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 230000008450 motivation Effects 0.000 description 1
- 230000007935 neutral effect Effects 0.000 description 1
- JTJMJGYZQZDUJJ-UHFFFAOYSA-N phencyclidine Chemical compound C1CCCCN1C1(C=2C=CC=CC=2)CCCCC1 JTJMJGYZQZDUJJ-UHFFFAOYSA-N 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 238000012018 process simulation test Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000007670 refining Methods 0.000 description 1
- 230000008929 regeneration Effects 0.000 description 1
- 238000011069 regeneration method Methods 0.000 description 1
- 230000002040 relaxant effect Effects 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 238000005096 rolling process Methods 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000000528 statistical test Methods 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000009897 systematic effect Effects 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
- 238000010200 validation analysis Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
- 230000003442 weekly effect Effects 0.000 description 1
- 238000010626 work up procedure Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/38—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/067—Enterprise or organisation modelling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/01—Social networking
Definitions
- This application relates to data analysis, and more particularly to a sociological data analysis.
- This application addresses an invention to substantially improve the complex effort of responding to a discovery request, and the demands of performing an investigation.
- the two halves, which are often performed in parallel we will call review and investigation respectively.
- a common review method is when reviewers annotate items with one or more tags indicating how the content should be categorized. Based on these reviewer categorizations, each item either produced to the counter party, or noted in a privilege log but (generally) not produced, or nor produces because of irrelevance to the discovery request.
- the traditional process of handling a discovery request is time and labor intensive, and as a result has a high cost. Furthermore, it is extremely difficult to obtain consistent and accurate results amongst reviewers which is a significant problem in itself, but especially when there are a large number of reviewers working to meet a discovery request.
- categorization requirements are varied and can include “hard” constraints such as conformance to relevant date ranges or custodial ownership, as well as broad references to a general topic—and all points on the continuum in between.
- Short format items such as email responses or IMs can be sufficiently lacking in content that they require other related items—such as those identified by discussions, in order to accurately assign any meaning to them.
- a first step of handling a discovery request often involves an investigative effort where the party served with a discovery request is interested in making its own conclusions about the matter at hand. It is often important for both review and investigation tasks to be done in parallel for the simple reason that the investigation effort may in some instances dictate that a case should simply be dropped, or that an attempt should be made to settle it based on “bad fact patterns.” While review and categorization of individual items is necessary in order to determine which items must ultimately be produced, it is a much different task than trying to analyze the collective meaning of the data.
- Analyzing corporate data for its meaning can quickly provide information about exactly what happened, and who might be important to an investigation effort.
- the present invention provides visualization, analysis, and a powerful query engine for many dimensions of actor behavior, with special attention given to how these different dimensions change over time, and may be correlated to one another.
- factors such as the emotive tones present in communication, and the apparent avoidance of written communication media are analyzed and visualized.
- a method to enable improved analysis and use of sociological data comprising identifying causal relationships between a plurality of documents, identifying a plurality of characteristics of a communication, including a modality used, actors involved, proximate events of relevance, and enabling a user to query based on the available characteristics.
- FIG. 1A is a flowchart displaying one embodiment of a high-level system overview.
- FIG. 1B is a screenshot of one embodiment of the main navigation window.
- FIG. 2 is a diagram of one embodiment displaying key concepts related to sociological data mining.
- FIG. 3A is a screenshot of one embodiment of a user interface design to represent discussions.
- FIG. 3B is a screenshot of one embodiment of timeline shaded in vertical sections by color to indicate nights and days.
- FIG. 3C is a screenshot of one embodiment of a vertical timeline color display indicating start and end of relevant time interval.
- FIG. 3D is a screenshot of one embodiment of a discussion showing a PDA icon.
- FIG. 3E is a screenshot of one embodiment of an audit trail.
- FIG. 3F a screenshot of one embodiment of a cluster header at the discussion and message levels.
- FIG. 3G is a screenshot of one embodiment of displaying Cluster contents.
- FIG. 3H is a diagram of one embodiment of how a user can extend the query to search for any potential variance of a document.
- FIG. 4A is a screenshot of one embodiment of an Actor Information Report.
- FIG. 4B is a screenshot of one embodiment of an Actor Information Report.
- FIG. 4C is a screenshot of one embodiment of an Actor Information Report.
- FIG. 4D is a screenshot of one embodiment of an Actor Information Report.
- FIG. 4E is a diagram of one embodiment of the different components available in an Actor Information Report.
- FIG. 5 is a flow chart of one embodiment of the mechanism used to determine whether an individual control is set for an item.
- FIG. 6A is a diagram of one embodiment of the behavior of a discussion when it contains items with different control settings.
- FIG. 6B is a flow chart of one embodiment of how adding new data affects existing discussions.
- FIG. 7 is a screenshot of one embodiment of a time-lapsed presentation.
- FIG. 8 is a screenshot of one embodiment of a thumbnail presentation options.
- FIG. 9 is a diagram of one embodiment of different event types.
- FIG. 10 is a screenshot of one embodiment of an Item Report.
- FIG. 11 is a diagram of different types of communication graphs.
- FIG. 12 is a screenshot of one embodiment of a communication graph displaying discussions.
- FIG. 13 is a screenshot of one embodiment of a communication graph displaying the capability of performing graphical querying.
- FIG. 14 is a flowchart of one embodiment of the use of graphical query to get query results or query controls.
- FIG. 15 is a screenshot of one embodiment of a communication graph displaying discussions annotated with a phone icon to represent a phone call.
- FIG. 16 is a screenshot of one embodiment of a communication graph displaying an icon for a N way phone call.
- FIG. 17 is a screenshot of one embodiment of a communication graph displaying discussions annotated with phone icons that designate whether phone records are available for the phone event or not.
- FIG. 18 is a screenshot of another embodiment of a communication graph displaying discussions annotated with phone icons that designate whether phone records are available for the phone event or not.
- FIG. 19 is a screenshot of another embodiment of a communication graph displaying discussions annotated with phone icons and mouse over capability that provides relevant additional information about the phone call.
- FIG. 20 is a screenshot of one embodiment of a communication graph displaying discussions using different line styles and phone icons to designate whether a particular discussion has a phone event and whether phone records are available for the event.
- FIG. 21 is a screenshot of another embodiment of a communication graph displaying discussions using different line styles and to designate whether a particular discussion has a phone event and whether phone records are available for the event.
- FIG. 22 is a screenshot of another embodiment of a communication graph displaying only the communications involving an actor selected by the user.
- FIG. 23 is a flow chart of one embodiment of different instruction types.
- FIG. 24 is a screenshot of one embodiment of the graph of instruction relaying.
- FIG. 25 is a screenshot of another embodiment of the graph of instruction relaying displaying one embodiment of highlighting direct instructions by the use of a darker line.
- FIG. 26 is a screenshot of another embodiment of the graph of instruction relaying displaying another embodiment of highlighting direct instructions by the use of an icon.
- FIG. 27 is a screenshot of another embodiment of the graph of instruction relaying displaying rings for mere forwards and explicit instructions.
- FIG. 28 is a screenshot of one embodiment of displaying actor proximity for both professional and personal communications.
- FIG. 29 is a screenshot of one embodiment of displaying emotive content for communications between actors.
- FIG. 30 is a screenshot of another embodiment illustrating the capability of graphical query.
- FIG. 31 is a screenshot of one embodiment of showing changes in tone over time.
- FIG. 32 is a screenshot of one embodiment of displaying actor proximity by the number of used contact channels.
- FIG. 33 is a screenshot of one embodiment of a graph based on discussions.
- FIG. 34 is a screenshot of one embodiment of communication in the context of the organization chart displaying a missing link and communication around the organization chart.
- FIG. 35 is a screenshot of one embodiment of communication in the context of the organization chart displaying communication boundaries.
- FIG. 36 is a screenshot of one embodiment of a graph displaying the spread of information.
- FIG. 37 is a screenshot one embodiment of sequentially displaying mixed type discussions.
- FIG. 38 is a screenshot of one embodiment of displaying probability of unrecorded event.
- FIG. 39 is a screenshot of one embodiment of displaying relevant comparative data about discussions including discussion length and the number of discussions that ended in a “call me” event.
- FIG. 40 is a screenshot of one embodiment of sequentially displaying mixed type discussions along with pivotal events.
- FIG. 41 is a screenshot of one embodiment of a tonal analysis of Actor communication.
- FIG. 42 is a screenshot of one embodiment of a tonal analysis of actor to actor group communication.
- FIG. 43 is a diagram of one embodiment of how a sentence is analyzed for tonal content.
- FIG. 44 is a screenshot of one embodiment of a tonal analysis of actor communication illustrating an icon for quoted content.
- FIG. 45 is a screenshot of one embodiment of an analysis of actor communication illustrating emotive content.
- FIG. 46 is a screenshot of one embodiment of an analysis of actor communication illustrating the gauge for negative tonal content.
- FIG. 47 is a screenshot of one embodiment of an analysis of actor communication illustrating the clustering presentation method.
- FIG. 48 is a screenshot of one embodiment of emotive content.
- FIG. 49 is a screenshot of one embodiment of Actor Heartbeat.
- FIG. 50 is a screenshot of one embodiment of potential tampering with backups of archival formats.
- FIG. 51 is a screenshot of one embodiment of document lifecycle view.
- FIG. 52 is a screenshot of one embodiment of document lifecycle view in comparison to ad hoc workflow.
- FIG. 53 is a diagram of one embodiment of how a burst of activity is determined for a document lifecycle.
- FIG. 54 is a screenshot of one embodiment of an Actor Information Report.
- FIG. 55 is a diagram of one embodiment of how a document's ancestral lineage is determined.
- FIG. 56 is a screenshot of one embodiment of privilege log.
- FIG. 57 is a screenshot of one embodiment of intersection of review decisions and clusters.
- FIG. 58 is a diagram of one embodiment characterizing clusters into subsets.
- FIG. 59 is a diagram of one embodiment of how a corruption problem is handled by the system.
- FIG. 60 is a screenshot of one embodiment illustrating a corrected item which was formerly corrupted.
- FIG. 61 is a screenshot of one embodiment of illustrating how to correct custodial information.
- FIG. 62 is a screenshot of one embodiment of illustrating how to correct custodial information.
- FIG. 63 is a diagram of one embodiment of correcting actor information.
- FIG. 64 is a screenshot of one embodiment of actor highlighting.
- FIG. 65 is a screenshot of one embodiment of query controls used to determine who has seen which items.
- FIG. 66 is a diagram of one embodiment illustrating rules to batch documents.
- FIG. 67 is a screenshot of one embodiment of individual batch assignment.
- FIG. 68 is a screenshot of one embodiment of bulk batch assignment.
- FIG. 69 is a screenshot of one embodiment of classification of review times.
- FIG. 70 is a screenshot of one embodiment illustrating percentage turnover of original ontology classification.
- FIG. 71 is a diagram of one embodiment illustrating how a system projects for review completion time.
- FIG. 72 is a diagram of one embodiment illustrating how an attachment is coded.
- FIG. 73 is a screenshot of one embodiment of folder tree view structure.
- FIG. 74 is a diagram of one embodiment of attributes of finding folders.
- FIG. 75 is a screenshot of one embodiment illustrating related content across findings.
- FIG. 76 is a diagram of one embodiment of how findings folders can be merged.
- FIG. 77 is a diagram of one embodiment of a system prompt to merge folders containing overlapping data.
- FIG. 78 is a screenshot of one embodiment of tabular reports of items in findings folders.
- FIG. 79 is a screenshot of one embodiment illustrating user overlap in findings folders.
- FIG. 80 is a screenshot of one embodiment of timeline view of items in findings folders.
- FIG. 81 is a screenshot of one embodiment of a system prompt to limit query to items in findings folders.
- FIG. 82 is a diagram of one embodiment of prioritizing findings folders using user-assigned ordinal priorities.
- FIG. 83 is a screenshot of one embodiment of a highlighter notes panel.
- FIG. 84 is a flow chart of one embodiment of a mechanism to determine whether a document is reviewed “interesting” for the same reasons or different reasons.
- FIG. 85 is a screenshot of one embodiment of profiling findings folder contents.
- FIG. 86 is a screenshot of one embodiment of “is important” attribute of a findings folder.
- FIG. 87 is a screenshot of one embodiment of mapping the findings to a decision tree.
- FIG. 88 is a screenshot of one embodiment of query controls.
- FIG. 89 is a screenshot of one embodiment of displaying discussion information in a tabular report form.
- FIG. 90 is one embodiment of a panel of user interface icons.
- FIG. 91 is a screenshot of one embodiment of a timeline within a date range.
- FIG. 92 is a screenshot of one embodiment of a timeline within a date range depicting when communications via email, instant message and phone call occurred.
- FIG. 93 is a screenshot of one embodiment of timeline depicting discussion matches by type.
- FIG. 94 is a screenshot of one embodiment in which search results across findings are displayed in a Venn diagram.
- FIG. 95 is a diagram of one embodiment of options for display of quoted text.
- FIG. 96 is a diagram of one embodiment illustrating the relationships between concepts related to automated production.
- FIG. 97 is a flowchart of one embodiment of the automated production process.
- FIG. 98 is a diagram of one embodiment showing a lattice, and an unordered and reordered matrix.
- FIG. 99 is a screenshot of one embodiment of the automated production analytical tool.
- Actor [ 205 ] The human being associated with one or more distinct electronic identities [ 255 ]; for example, different email accounts, IM handles, system logins, etc.
- An actor [ 205 ] may be deemed to have more than one personality [ 250 ] if the content created or received by at least one of the different electronic identities [ 255 ] varies significantly from that of the others.
- Actor Heartbeat [ 220 ] A base line of the set of electronically recorded actions that an actor [ 205 ] typically performs when working. Among other things, this is used to determine when specific actors [ 205 ] were not working, or not working as they normally do.
- Discussion [ 235 ] A possibly heterogeneous partially ordered set of electronic record items for which it is presumed that any item is causally related to all items immediately following it.
- partially ordered we mean that there is a relationship “ ⁇ ” between items with the property that, if L ⁇ M and M ⁇ N then L ⁇ N. If M ⁇ N, we say that N follows M. When we say that N immediately follows M, we mean that M ⁇ N, and that there is no item L such that M ⁇ L ⁇ N.
- IM instant message
- Text blocks [ 225 ] A contiguous block of text that is associated with one author, and is substantive enough to be treated potentially as an independent object in the system.
- the user interface features object-centric views of the most important types of data.
- the most complex of these is the discussion template [ 105 ] which allows the different items [ 125 ] of the discussion to be viewed and coded (that is, tagged with appropriate values) by the user. This is illustrated in FIG. 1A .
- the discussion template [ 105 ] also provides controls and displays attributes for the container object of the discussion itself. From the discussion template [ 105 ], the user may navigate to views of most of the other types of objects in the system. These objects may include, but are not limited to: Actor Information Reports, Communication Graphs that interrelate actors to one another, Event & Attendee Reports.
- an Audit Trail screen provides a fully integrated view of all knowledge the system has about a discussion and each of its constituent objects. This includes, but is not limited to; its current state in the workflow system, which users have moved, foldered, flagged, annotated or reviewed the item and when, which types and instances of folders it currently exists in, tags assigned by the system such as the topic(s) or emotive tone(s) or foreign languages that are contained within each item, and different presumed versions of the item.
- tags assigned by the system such as the topic(s) or emotive tone(s) or foreign languages that are contained within each item, and different presumed versions of the item.
- the user interface is presented with conventions common to internet browsers.
- central navigation is provided through a navigation bar which is always present. Its location varies with embodiment, but in most embodiments it appears either horizontally on the top of page or vertically on the left-hand side of the page.
- FIG. 1B when the user launches the application and has authenticated, a main navigation window is provided as illustrated in FIG. 1B . As pictured in FIG. 1B , in one embodiment, the main function of this window is to facilitate the user making an initial selection from the many task areas provided by the system. Because of the large number of visualizations and reports provided by the system, for most types of objects a default view type can be configured, in most embodiments. In some of these embodiments, these defaults may be attached to specific matters, for example, a specific investigation, rather than being global in nature. This is because what aspects of the data are most important to understand may vary greatly with the specific scenario at hand.
- the user may select the “Set Preferences” button [ 165 ] in order to display, select and scope the available preferences.
- “Set Preferences” button [ 165 ] may be used.
- data may be presented in windows, frames, or alternative display mechanisms may be used.
- the “Review Documents” button [ 125 ] if selected will take the user to either a set of batches of documents that she has been assigned to review, or to controls that allow her to assign batches, either to herself or to other users of the system. Configured preferences, and whether the user in question has administrative privileges determines which.
- the “Review User Performance” button [ 140 ] if selected will, depending on configured preferences, bring up reports on reviewer performance. In some embodiments, it may instead bring up a panel with a list of links to reports. In most embodiments, if the user does not have administrative privileges, this button will not appear or will not be available for selection.
- the research-oriented buttons [ 130 , 135 , 145 , 150 ] can be configured to bring up a particular window of the user's choosing. As will be discussed in subsequent sections, in most embodiments the system can generate many varieties of windows which are divided between a specified set of query controls and a resulting visualization once a query has been executed. If the user decides that she wishes to research actors, clicking on the “Research Actors” button [ 145 ] can bring up a window with query controls featuring a list of the actors of greatest interest for her to choose from and which will return results in the user's preferred visualization. Examples of the many variations available will be provided in subsequent sections.
- buttons do have default views that they are associated with.
- the “Research Actors” button [ 145 ] may bring up a panel that allows the user to select one or more actors, or actor groups. Once he has done so, the Actor Information Reports for these actors will be generated and displayed.
- the “Research Events” [ 135 ] may bring up the “Events & Attendees” report
- the “Research Documents” button [ 150 ] the Document Lifecycle View.
- the “Your Folders” section [ 170 ] of the screen displays the user's personal folders of different types.
- “Your Messages” [ 175 ] contains any notifications sent by the system, as well as messages broadcast by other users of the system.
- “Your Reports” [ 180 ] contains a list of reports that were generated at specified intervals as requested by this user.
- the “Select Matter” button [ 155 ] allows the user to limit the current session to one or more particular matters, for example a specific investigation or other project. Selecting a particular matter will have the effect in most embodiments of limiting the information available for display. For example, if one or more particular matters are selected, actors who are not relevant to these matters will not be available in the system, and will not be visible in any way to the user. Note that since distinct investigations may often overlap with one another, there may be instances where selecting more than one matter is highly useful. If all of the projects that a given user is working on are interrelated in some fashion, by not selecting a matter, in most embodiments, the user will see all data related to the different matters as if it were a single collection of data.
- FIGS. 3A-G represent one embodiment of a discussion template.
- Some of the features available in the discussion template in one embodiment include: a timeline view [ 310 ], a special toggle [ 308 ] will open or hide the global controls [ 309 ], and a “duplicate” indicator [ 333 ].
- the timeline view [ 310 ] is one way of viewing a discussion. As pictured here in one embodiment, the timeline view [ 310 ] is displayed in the top frame of the window.
- the timeline view [ 310 ] depicts in parallel different kinds of activities which include but are not limited to email, IM, SMS, phone call activity, journal entries, notes and calendar entries. Most of these activities correspond to individual items in the discussion, in most embodiments. However, not all. For example, key external events of importance that transpired during the lifespan of the discussion may also be rendered in this view in some embodiments.
- each of these different kinds of activities is represented by a box [ 326 ] which upon mousing over, or in some embodiments, clicking, provides more information on the activity.
- each kind of activity is depicted on its own display line [ 311 ] in order to be able to display concurrent events, such as IM's that are being sent during the course of a meeting.
- different shapes or visual representations could be used to indicate the presence and relative length/strength/significance of an item. In some embodiments, clicking on the shape causes the frame to make the selected item the topmost one.
- the width of the boxes [ 326 ], in one embodiment, depicts the length of time over which the activity took place. For example, in FIG. 3A one can see that the IM activity lasted for about an hour out of total of four hours for the entire timeline [ 310 ].
- there is a control which allows the user to toggle to see all traffic among the parties in question during the timespan of the discussion.
- the time zone used in the timeline view [ 310 ] will default to that of the majority of primary actors in the discussion. In some embodiments, if primary actors are in different time zones, an indicator will appear in the timeline view [ 310 ] to call this fact to the user's attention.
- the timeline is shaded [ 324 ] in vertical sections by color to indicate days, nights, and weekends and holidays, as illustrated in FIG. 3B . In some of these embodiments, weekends are drawn in a different color than are holidays. In some embodiments, other visual indicators including, but not limited to patterns, shapes, backgrounds tiled with images and animated effects are used to indicate days, nights, weekends and holidays in the timeline view.
- the concept of “day”, “night”, “weekend” or “holiday” are calculated on the basis of characteristics of the primary actors in the discussion. These include, but are not limited to: what time zone and country they are in, their standard hours of work as can be determined by the relevant statistics from the Actor Information Report, and their standard days of working. In other embodiments, the computation may be done for all actors in the discussion, or all actors in the corpus, or may be defined by the user.
- vertical lines may be displayed to indicate the start and stop of intervals of interest with respect to the document review effort, for example, for different claims in a subpoena, or of the subpoena itself. This is illustrated in FIG. 3C .
- investigations are often limited to a specific interval of time. If a document lies outside of this interval of time, it may not need to be looked at all, or reviewed against as many different criteria. Multiple pairs, or even unpaired and start and stop lines [ 325 ], [ 328 ] are therefore permitted in most embodiments.
- the start lines [ 325 ] are green which indicates the starting date [ 327 ] and optionally, time; the ending lines [ 328 ] are red which indicates the ending date [ 329 ] and again optionally, time. Both may be labeled with brief labels, and/or have more descriptive labels that may be seen via mouseover. Further, in some embodiments of the invention, these subpoena cut-off indicators [ 325 , 328 ] are used in conjunction with workflow rules that provide a warning to the user that a review decision may be inappropriate on the basis of timeline. In one embodiment, the system may then bring the timeline view [ 310 ] into focus.
- Attention may be called to the relevant timeline [ 310 ] cut-offs [ 325 , 328 ] by animating the line, highlighting it, circling it, or otherwise visually calling attention to it.
- an arrow or other vertical divider could be used instead of a line.
- This timeline [ 310 ] representation provides various advantages. It shows how different types of time spanning objects overlap with one another. For example, if a meeting lasted for 4 hours, during which time 3 emails were sent, and 10 rounds of 1M were exchanged among the relevant actors, this fact could very easily be seen by a quick look at this view. By contrast, a view that showed each object as a box of approximately fixed width provides much less information. In that view, a meeting that was scheduled to last a week does not visually appear longer than a five minute IM conversation. Furthermore, it is difficult to see what else occurred over the weeklong meeting, since the object representing the meeting would not visually appear to span a week.
- This view is visually compact enough that it can reasonably be displayed in the same window as the rest of the transcript view [ 313 ] so that it is immediately present to provide guidance.
- the timeline view [ 310 ] may be presented in a separate window, or frame.
- Some embodiments of the discussion template may feature slightly different embodiments of the timeline view [ 310 ]. Further, in many embodiments of the system, the timeline view [ 310 ] is widely used in different kinds of query results, both to support graphical querying, and to more vividly illustrate the time intervals in which the relevant communication events were occurring.
- individual item controls [ 316 ] may be hidden or viewed via a special toggle button [ 322 ] on an item by item basis, or globally.
- item controls [ 321 ] will be displayed for any item whose state is in some way different with respect to the settings of these controls [ 321 ] from the parent discussion.
- the individual controls [ 321 ] include a control that allows the setting of an individual item to be used for all items; in other embodiments, the same control promotes the current item's state to that of the discussion as a whole (from which other current and future members will inherit.) This is useful because often a reviewer may be tempted to use the global controls [ 309 ] at the top of the screen to code the whole discussion, based on what she initially sees of the first item or two. However, especially in the case of long discussions, there may be some information much further down that retroactively changes the reviewer's impression of all that has gone before.
- the location of the controls to code individual items may be located anywhere visually apparent to the user—in one embodiment, the controls [ 321 ] are located to the left of the content.
- some embodiments will support document type-specific controls. This may be necessary, for example, because a subpoena might not include a requirement to produce certain types of items, which may nevertheless end up in a discussion [ 235 ]. In such an instance, there might be no controls [ 321 ] made available for these items. Conversely, there could be additional controls [ 321 ] for certain types of items which might be desirable, but which would not have further applicability—for example, a checkbox to indicate whether a speech-to-text transcript of a phone call was decipherable. In most embodiments, in this latter case, the discussion cannot be moved along in workflow until all additional required item controls [ 321 ] have been set. In some embodiments, if a discussion contains multiple items of a type that require additional control(s), the control(s) in question will be dynamically added to the global controls [ 309 ].
- the user is allowed to select different blocks of text from within the transcript view [ 313 ].
- the selection brings up a panel where the user may indicate that the presence of this particular text causes any item that contains it to be coded in a certain way. For example, as responsive to specification 1 of a particular subpoena. As a result of such coding, all documents which contain the relevant text are coded as responsive.
- the scope of this action is restricted to the current discussion, but others may allow different scopes, including related discussions, all items pertinent to particular matters, or even the whole corpus.
- a special toggle [ 308 ] will open or hide the global controls [ 309 ] for the parent discussion. This is done to conserve space, as these controls could in some cases consume a considerable amount of screen real estate. In some embodiments, these controls are collapsed automatically if there are items in the discussion that have, or are inferred by the system to have, heterogeneous review properties.
- a “duplicate” indicator [ 333 ] is added to show which attachments—or other discussion members—are duplicates, near duplicates, or prior or subsequent versions of one another. In one embodiment, the following variations are provided for with their own icons; other embodiments could have additional ones as illustrated in FIG. 3F .
- the term “duplicate” in one embodiment includes one or more of the following:
- this “duplicate” information also is displayed in the header information [ 332 ] for the discussion.
- This may include a visual indicator [ 333 ] as indicated in FIG. 3F , which may be clicked, in one embodiment, in order to navigate to other related items.
- clicking on this control [ 333 ] brings up a panel [ 339 ] which contains information about the applicable category of related documents, and controls to code them.
- this includes a tabular view [ 341 ] with links to all related documents, such as different versions of the same document, or items that are for whatever reason in the same cluster(s).
- each document there is an icon [ 342 ] associated with each document that indicates the more exact nature of its relationship to the parent document, for example that it is a “general near duplicate.”
- global coding controls [ 309 ] allow the bulk coding [ 340 ] of all items in the cluster. This is pictured in FIG. 3G .
- each item in the cluster also has individual coding controls.
- items are designated by a different color and/or visual indicator based on the nature of their relationship to the selected item, as shown in the legend [ 342 ].
- related items that also belong to the current discussion are also designated by some type of different color and/or visual indicator.
- this panel [ 339 ] provides links to one or document lifecycle views that are associated with the displayed items. This is a view which displays the different iterations and versions of an item that came into existence over the course of time.
- One embodiment of the document lifecycle view is shown in FIG. 51 .
- This collection of features provides a huge time savings advantage for the user, because, for example, if the same 200-page document shows up 4 times in the same discussion [ 235 ], she needs now only review it once.
- this feature allows different kinds of near-duplicate items to be reviewed at the same time, thereby saving time.
- the system may perform a “diff” (showing differences between items) of these near duplicate items against one another, in order to further facilitate this task.
- each of the types of indicators noted above has corresponding operators in the query language.
- additional related operators exist. For example, in one embodiment there are separate operators for prior presumed version versus subsequent one, and descendants from true widely used boilerplate versus the case in which different branches from the same document develop over time. This enables a user to select a document, and search for any of the potential variants of the document, collectively, individually, or by any categorization.
- One embodiment of the query operators is pictured in FIG. 3H .
- an evidence type indicator [ 317 ], and a day/night indicator [ 343 ] are added to the transcript view [ 313 ]. Clicking on the day/night indicator [ 343 ] brings up a panel that provides information as to the normal working day of the primary actors in the discussion when the indicator is in the discussion header [ 332 ], and the actors associated with the particular item when the indicator [ 343 ] appears in an item header.
- this information is obtained by averaging the start and end times of business-related communication on working days.
- this information is obtained using other techniques including, but not limited to, median of start and end times of business related communications, or other metrics that are captured in different embodiments of the Actor Information Report.
- the system uses information from human resources database to determine a particular actor's working day, which may include when the employee timed in, entered their physical place of work with a security card, or logged into a particular system.
- a particular actor's working day may include when the employee timed in, entered their physical place of work with a security card, or logged into a particular system.
- Clicking on the evidence type indicator [ 317 ] brings up an explanation as to why that particular item was brought into the discussion, for example, that it made reference to a document that was attached to a prior email, and also involved the same actors. This is useful because under certain circumstances, attorneys might have to testify as to how specific conclusions were drawn. In the prior art, no such controls were available in the transcript view [ 313 ].
- a navigation control [ 315 ] which provides a means of navigating from one item to the next with a single click, thereby enabling directed navigation which is both faster and less stressful than using scrolling;
- a display control [ 308 ] which provides the ability to collapse or hide the global controls [ 309 ].
- FIG. 3A An additional display is shown in a separate Actor Information tab [ 303 ] in FIG. 3A , to provide statistical information for primary actors in the discussion. (Some embodiments may provide this information for all actors, not just the primary ones.) This is known as the Actor Information Report. An exemplary embodiment of this report is pictured in FIGS. 4A-C . This screen may be reached, in one embodiment, by clicking on any occurrence of an actor's name, in which case the window will automatically move to the information for that actor, or the report may be navigated to directly by clicking on the Actor Information tab [ 303 ].
- the neighborhood graph (which may be reached via the Neighborhood tab [ 304 ]—includes a neighborhood graph [ 1105 ] for each primary actor.
- the neighborhood graph [ 1105 ] is a communication graph [ 1120 ] depicting the communication from one or more selected actor(s)—in this case, those in the current discussion—to those closest to these selected actor(s).
- “Closeness” may be defined by a metric selected by the user, for example, N handshakes out from the selected actor(s). Otherwise put, a neighborhood of radius r around a specified set of actors includes all actors that can be reached by following no more than r links.
- any other type of communication graph [ 1120 ] may appear as well, or may be specified as a preference on the part of the user.
- One embodiment provides the user with user interface controls that allow her to select the style and characteristics of communication graph [ 1120 ] that she wishes to see. (Communication graphs are explored in greater depth in a following section.)
- these two actor-related tabs [ 303 , 304 ] are combined into one tab. These additions are primarily useful in an investigation context, where it is advantageous to be able to quickly get a quick appraisal of the basic status and general behaviors of each primary actor in a discussion, so as to easily determine whether the currently examined interaction falls within normal behavior patterns or not.
- the discussion template includes but is not limited to these features: a name associated with a discussion [ 302 ], special toggles [ 314 ] to display (or not) different portions of the individual discussion items. These include, but are not limited to: “show/hide message content”, “show/hide header information”, “show/hide quoted text.” Other important information about the discussion and its constituent items are included in the discussion header [ 332 ] an exemplary implementation of which is shown in FIG. 3F . This includes, but is not limited to: primary actor identification, participants, the different item types present, the number of attached items, and the date range covered by the discussion.
- the header information for individual items contains item-level information, including but not limited to the item's name and unique ID within the system, information about any attachments and their document type and size, their item number within the discussion, the actor(s) related to the item, a time/date stamp, the evidence type indicator [ 317 ] and the day/night indicator [ 343 ].
- the window in one embodiment also contains buttons that allow users to perform basic operations, such “submit”, “reset” or “close”
- indicators may appear in the item header to indicate the media used for the communication.
- FIG. 3D illustrates such an indicator for a PDA [ 330 ].
- all changes to the status of any discussion or individual item are recorded and may be viewed in the audit trail log as illustrated in FIG. 3E .
- discussions may be globally coded by using user interface controls [ 309 ].
- One embodiment of the user interface controls [ 309 ] is shown in FIG. 3A .
- these user interface controls [ 309 ] are positioned at the bottom and/or top of the window.
- a user can indicate that all N items in a discussion are to be treated in the same way, for example, as all non-responsive. This makes for vastly faster review, because in most cases it greatly reduces the number of user actions that are needed to complete review of the documents in a discussion.
- FIG. 5 is a flowchart of one embodiment of determining whether individual controls should be displayed. These differences could include decisions made by other users as well as by the system.
- Some embodiments visually distinguish between the case when the user has made the decision, and the case in which the system has. For example if the system were to determine that some items were likely privileged and others not, the individual controls/indicators [ 321 ] would appear for the items that did not share the setting of the parent discussion, as a visual cue to the user. Other embodiments may include other visual indicators instead of, or in addition to this. Generally the global setting is determined by the state of the majority of items in the discussion.
- the system allows users to display or hide the quoted text in emails, either throughout a discussion or in an individual item. Especially for users who choose to display the quoted text, there is often an advantage to displaying the items in reverse chronological order, since the first message will likely provide an overview of at least some of the other messages. For all other uses, a chronological rolling out of all exchanges between the parties, irrespective of whether the text came from the original email, or only from a quoted textblock in another email, may be more advantageous. Note that most embodiments actually are displaying discussions via a topological sort; rather than a full ordering of matters if the discussion has different branches.
- this display is further augmented by allowing the content from each actor, including quoted text, to be individually displayed or hidden by clicking on a show/hide toggle control.
- the show/hide toggle control is of the same color used to represent that actor in that discussion (see section on “Configurable Highlighting of Actor Names.”)
- all textblocks from the same author, including quoted text will be displayed with the background color designated for that actor in the discussion, perhaps at a reduced saturation in order to promote readability.
- the font color could be used instead for this purpose.
- quoted text for which the originating email no longer exists will still be displayed even if quoted text is hidden. In most of these embodiments however, such text is rendered in a different color, or with some other visual indicator to indicate its special status. In most of these embodiments, the first instance (if there is more than one) of this quoted text will be the one that gets rendered. In other embodiments, as described in Sociological Data Mining Application the system may attempt to reconstruct the missing original message.
- FIGS. 4A-D present embodiments of a report format for actor information. However, different embodiments may present somewhat different information.
- the information itemized in FIGS. 4A-D can be thought of as a detailed workup of measurable aspects of communication for an actor.
- the system in one embodiment, provides the maximum possible number of dimensions in order to better detect aberrations (if any) from the standard behavior of the particular actor. This is because aberrations from standard behavior, even subtle ones, can indicate guilty knowledge, or other things of great interest from an investigation standpoint.
- the user can navigate with a single click to a large number of views for other types of objects that are related to the actor(s) featured in the report.
- An exemplary set of data available from the Actor Information report are illustrated by FIG. 4E . The closer integration in the user interface of this information helps users to more easily understand individual discussions in the context of the broader behavior patterns of the relevant actors.
- actor information reports for relevant actors appear in the actor tab in the discussion template.
- the report also is accessible from most query result views that contain actor object results.
- the Actor Information Report is the default view type configured for the actor object.
- the system will endeavor to correctly associate human actors with their various electronic identities.
- the Sociological Data Mining Application describes an embodiment of creating such associations. As noted elsewhere in this application, this task cannot always be done correctly by either human or machine. For this reason, different embodiments will vary in their presentation of actor identity information. For example, some will require that identity assignments be confirmed by a human administrator, while others will utilize, but enable correction, of the system assigned identity. Still others will list partial matches—that is cases in which the system has identified the possibility, but not the certainty, of the match of an identity to an actor—and optionally await user feedback.
- Primary email account in most embodiments refers to the email account that is associated with the actor's employer. However in other embodiments, it may simply refer to the email account from which the greatest amount of content emanates over the period of time of interest. In other embodiments, other metrics, including but not limited to number of e-mail sent or received, topics and content discussed in e-mail sent or received could be used to determine a “primary e-mail account”
- “Fastest response” response behavior in one embodiment is used to track normal user behavior.
- the basis of the metric for fastest response to email is not literally based on the average interval of time taken to respond, but rather the order in which the response occurs. For example, if when Jane comes into work in the morning, she often finds that she has received 100 emails from assorted actors since she last checked, the order in which she generally opens, and then responds to these emails can be used to adduce the relative importance of the actor to her. In some embodiments therefore, the number of emails from known actors that are skipped over in order to first answer an email from a particular actor is the basis for the metric of speed.
- Rapid fire response behavior can be determined by evaluating the behavior of the actor over a sufficient period of time to observe whether or not she responds to multiple messages after she has missed a few turns, or whether instead, she provides a summary response to only the most recent. For example, if three email messages are sent on a given topic while the actor is absent, if he responds to more than the last of these messages, it is considered “rapid fire” behavior. Note however that the same logic can be applied to any type of message, including but not limited to IM's and voice mails.
- the Actor Information Report contains a sub-report breaking down all topically categorizable content that was generated and/or received by the actor in question over user-selected period(s) of time. Most of these contain pie or other charts. Such “mindshare” reports may be useful in the construction of what some attorneys call a “good heart, empty head” defense; otherwise put, that the actor in question demonstrably paid so little attention to particular topic(s) that he could not reasonably be expected to have substantive knowledge of them. In some of these embodiments, any references by other actors to the selected actor having knowledge of the relevant topics are also included in the “mindshare report.”
- Some embodiments may perform and display correlations of the different dimensions of behavior that are being analyzed. As pictured, for example, a topic “FAS131” is associated with a predominantly negative tone.
- a discussion may be “exploded.” In other words, some items of a discussion may end up in different stages in a workflow.
- FIG. 6A One embodiment of these different options can be seen in FIG. 6A . This occurs for example when some of the items in a discussion are determined to be responsive while others are not, and the responsive items move to a subsequent review stage whereas the non-responsive ones go to a final bin.
- the discussion is broken down into as many separate chunks, or individual pieces, as are needed so that each individual item ends up in the correct spot in workflow for the settings that the user has assigned to it; from a workflow perspective, the discussion ceases to exist.
- a visual indicator appears in the header area that may be selected to view the entire logical discussion, regardless of where its various parts currently are in the workflow.
- items always remain displayed in their discussions, but depending on which location in workflow the discussion is currently being viewed in, the controls for certain items (those that are not currently in this particular location in workflow) will be disabled.
- a visual indicator appears that can be selected to see—and code—the item in its current location in the workflow.
- the controls for each item are displayed according to where each item currently resides in workflow. Since different workflow stages may have different controls associated with them, this means that the available coding controls [ 321 ] may be heterogeneous within a single discussion. In some embodiments, there is a control that allows previously “exploded” items to be pulled back into their original parent discussion.
- the full logical discussion can still be viewed outside of the context of workflow such as in a query results set.
- the newly added items are reviewed either individually, in some other group configuration, such as a cluster, or in sub-discussions built from within the newly added data, or from data that has not yet reviewed.
- a visual notification is added to the discussion to which the new items belong in order to indicate that new items have been added.
- the new items may be allowed to join their logical discussion.
- the whole discussion will be moved into a separate workflow area for review. In one embodiment, this is done only for discussions residing in user-specified locations in workflow. For example, users might wish to re-review augmented discussions that are currently in the “responsive” area but not in the “non-responsive” one.
- more than one of these techniques can be used, depending on the specific attributes of either or both the particular item(s) being added and the discussion that the items are being added onto. That is, for example, certain types of items may be added into the existing workflow, while others of potentially greater importance would be isolated.
- time slicing we mean creating equal increments of time within the time interval of interest specified by the user, and creating a snapshot image for each time increment.
- FIG. 7 depicts a thumbnail gallery [ 715 ] which provides a full-size snapshot [ 710 ] of the currently selected view in the main view.
- FIG. 8 depicts a thumbnail gallery implementation in which the thumbnail images [ 805 ] for different time slices are arrayed in a matrix configuration.
- the user may specify the length of time covered per thumbnail image; in some embodiments the user may specify the number of slices to create.
- the last time slice will simply be longer; in other embodiments, the “extra” is equally divided among the different snapshots, while in still other embodiments, a partial timeslice will be created at the beginning or end that is especially marked.
- the speed of display (or “play”) and the length of time increments are user configurable.
- the advantage to this type of visualization is that it allows the user to very quickly spot changes in patterns over time, in a manner that is much easier than scrolling or dragging a slider.
- a slide sorter [ 715 ] metaphor may be utilized in conjunction with this type of presentation.
- a slider [ 720 ] is available for manual navigation; the slider also moves automatically, displaying the current time interval while the view is “playing.”
- VCR-like controls [ 705 ] are provided as the primary means of navigation, however the user may also click on a thumbnail in the slide sorter [ 715 ] to restart the “movie” playing from that point (or to bring up that timeslice in the main view [ 710 ] when the window is not in “play” mode.)
- Some embodiments may choose to augment portions of the view of greatest interest—such as detected anomalies—with additional visual effects. These may include, but are not limited to: flashing, pulsing, change of color or brightness, jiggling, moving, or being surrounded by a colored frame—or are otherwise brought to the user's attention.
- these movie-like presentations are created by the system itself by shrinking or “thumbnailing” regular views of the desired timeslices of information, and then “playing” them sequentially. In other embodiments however, this may be achieved through the use of a third party application.
- Attributes of a meeting event [ 925 ] may include but are not limited to:
- a calendar event for it; more generally, whether or not it has been scheduled, or appears to be ad hoc in nature.
- it is useful to make a distinction between ad-hoc meetings such as a request to “discuss it over lunch” versus regularly scheduled periodic meetings for which a change in schedule can be easily observed, and may be of interest to observe—for example, an emergency board meeting that is scheduled 3 weeks earlier than normal.
- accompanying text or document for it including but not limited to: meeting minutes for a meeting, a speech to text transcript of a phone call, and/or other accompanying material to a meeting such as presentations, a packet of information, or action items list. If such documents do exist, in some embodiments, their properties become attached to the meeting object—for example the presence of negative tonal content.
- this information also is made available as part of the Actor Information Report.
- it will appear in its own tab in the templates for those discussions which contain such an event, which to say an event that has actor attendees within the universe of the corpus.
- this report exists as a subset of the actor information tab in the discussion template.
- it is accessible by double clicking on any component of the graphic representation of such event objects, such as that which appears in a timeline view.
- the query interface allows the user to specify that only event objects should be retrieved; in most embodiments whenever event objects are retrieved as part of a query result, they are decorated with a report icon that if clicked on will display this report. Note that in most embodiments, event objects that have supporting transcript documents can be retrieved in searches that hit keywords or other search terms in the supporting transcript document.
- the present system provides communication graphs [ 1120 ] in order to visualize further dimensions of communication behavior that are of value, particularly in an investigative context. These are graphs that depict actors as nodes that are linked together by arcs of different styles in order to indicate communication that has occurred between the actors, or related relationships. Some of these variations are focus-related. That is, they exist primarily to filter out less relevant information, for example by allowing the user to set a bi-directional threshold for the number or type of communications that must have been exchanged in a given time period in order to connect two actors together. We call this a threshold graph [ 1115 ].
- a target set of actors or allowing the user to select a target set of actors, and then rendering the links and nodes to the actors closest to these actors according to the parameters set by the user, which we call a neighborhood graph [ 1105 ].
- a “grapevine graph” [ 1110 ] which given a set of actors, renders nodes and arcs for only the social hubs of the network.
- FIG. 11 illustrates one embodiment of some of the basic variations.
- Some embodiments contain communication graphs which display actors connected by lines based on mutual participation in the same discussion.
- An example of this is shown in U.S. patent application Ser. No. 10/358,786, entitled “A METHOD AND APPARATUS TO VISUALLY PRESENT DISCUSSIONS FOR DATA MINING PURPOSES,” and assigned to the assignee of the present application.
- the system described extends this to include phone calls and in-person meetings.
- Different embodiments may support somewhat varying behaviors for mouseover and clicking, such as a simple pop-up on mouseover [ 1210 ] which displays the number of phone calls in the discussion.
- actors are depicted by an image [ 1205 ], often a photograph of the actor.
- the actors are depicted by other rendered images as selected by the user. This might include, but is not limited to, arbitrary icons, the names of different electronic identities used by the actor, an identifier used by the investigator for the actor, or various fields in the Actor information report. In this embodiment, each discussion is depicted by an individual line [ 1220 ].
- the canvas tools include a connector tool, which the user can use to indicate to the system that she believes that there is a relationship that exists between the actors.
- a connector tool which the user can use to indicate to the system that she believes that there is a relationship that exists between the actors.
- Such feedback from the user can be used by the system as a hint to connect “mystery” electronic identities to the right actor, or depending on the exact visualization, to indicate that there is evidence of a social relationship, or that there was a reporting relationship that for some reason was not entered into the system.
- FIG. 13 illustrates the idea of graphical query support; selecting a number of discussions by partially or fully including them in the marquee [ 1305 ] will specify a query whose results [ 1310 ] are indicated on the right-hand side of the window.
- graphical querying also may fill in query controls in addition to, or instead of displaying results.
- a small phone icon [ 1505 ] will appear at the base of the line, on the side of the actor who placed the call. One embodiment of this is depicted in FIG. 15 .
- both actors placed calls to each other both would be annotated with a phone icon [ 1505 ].
- the lines will connect all the actors involved to a separate icon [ 1605 ].
- FIG. 16 One embodiment of this is shown in FIG. 16 .
- clicking on phone icon will bring up all that is known about the call, including any wav or similar files (including speech-to-text output) that might be available, and any references to it, such as in an online calendar. Note that most embodiments will similarly treat multiple person meetings, but use a different choice of icon.
- a small phone icon [ 1705 ] in one embodiment a question mark (not shown), will be similarly placed.
- Other embodiments may just use a “?”. This is illustrated in FIG. 17 .
- Other embodiments may similarly separately designate presumed or possible phone events, but with slightly different visual effects, as illustrated in FIG. 18 .
- Different embodiments may provide different visual representations for each attribute of a meeting object. Some embodiments allow the user to select which attributes they wish to visualize, as well as how she wishes to visualize them.
- the line can be annotated with a phone icon to indicate that the discussion contains at least one phone call.
- the phone icon may be further annotated with a number that indicates the number [ 1910 ] of phone call items in the discussion.
- FIG. 19 One embodiment of this is illustrated in FIG. 19 .
- While some visualizations render each discussion as a separate line; others may condense similar discussions (that is, those that share relevant attributes) into the same line, but then annotate that line with a number, or adjust the width of the line to indicate the number of individual discussions which are represented by the line. Similarly for singleton items. Some visualization may zoom out to condense all discussions involving the same actors into one line. In one embodiment, clicking on or mousing on or near the line will bring up a floating box with the breakdown itemizing the different discussions and/or singleton items represented by the selected line.
- the meeting object is decorated with an icon wherever it appears in the user interface, and by clicking on the icon or some other control, the relevant text document may be brought up.
- the meeting object will be returned in user queries if a search term is found in the accompanying text document.
- the user interface allows the user to determine which of these types of discussions or relationships she wishes to see displayed at once. In some embodiments, if different types of discussions (including those with no meeting object items) exist between the same 2 actors, then separate lines [ 2105 ] for each different style would be drawn next to one another. An example of this is illustrated in FIG. 21 . In some embodiments, as shown in FIG. 22 , the user can click on an actor icon [ 2205 ] to get a thumbnail image [ 2210 ] which visualizes only the discussion(s) involving the selected actor. In most embodiments, this thumbnail image [ 2210 ] may be expanded by clicking on it.
- a communication graph can be constructed that indicates each time that an instruction [ 2305 ] was provided by the sender to the receiver.
- the system determines whether the instruction [ 2305 ] was a “real” instruction [ 2310 ], that is, one in which actionable instructions were provided, or a mere forwarding [ 2315 ] of information which leaves the recipient to determine the action that should be taken in response.
- well known linguistic techniques are used to distinguish pleasantries that are arguably instructions, but very vague ones, such as “please take care of this” from imperative forms that provides substantive and specific instructions—for example “Call Joe and ask him to hold the shipment of component X.”
- the specific or directly actionable (and hence meaningful) instructions [ 2310 ] can be identified on the basis of various techniques, including but not limited to, the presence of proper nouns, more complex sentence structures, and verbs that are statistically uncommon with respect to the current corpus.
- Constructing communication graphs for instruction relaying has significant application in determining how an organization is really being run, who is giving instructions—and who is unable or unwilling to provide real instructions [ 2310 ] in writing—and who is carrying them out or interpreting them. This in turn is often of importance in complex white collar crime investigations, since the “interpreters” of the instructions often prove to be the most powerful and hence interesting people. For this reason, visualizations often focus on those “instructions” that are not actionable or “mere forwards” [ 2315 ], and which require significant interpretation.
- Some embodiments will require a distinct, identifiable block of content within the parent communication (such as an email) against which the instructions are to be applied (and against which any additional content is to be measured.). These include, but are not limited to: a forwarded email, an attached email or any kind of attached document, a hyperlink of any kind, a text block, or an embedded OLE object.
- system identifies specific kinds of instruction language, which may be determined as needed for different industries.
- FIGS. 24 through 27 depict embodiments of a visualization of instruction graphs.
- FIG. 24 depicts one embodiment of a graph of “mere forward” instruction relaying.
- the user interface features a panning widget [ 2420 ] that results in the magnification of the screen area under it [ 2405 ].
- Arrows [ 2415 ] point in the direction of the actor to whom the not-quite-instructions or “mere forwards” are being relayed.
- the magnified portion of the graph appears in a separate panel [ 2510 ] that may be contained within the current parent window, or which in some embodiments may be spawned in a new window.
- Different embodiments may use different colored or otherwise indicated lines [ 2505 ] in order to distinguish messages that contain actionable instructions, from “mere forwards”, and those for which this type of analysis is not relevant. In one embodiment, this is determined based on the type of object—for example, the item is an email sent to a jokes list—as determined by ontological or statistical means.
- FIG. 26 Other embodiments may employ only one line color, but be annotated by an icon [ 2605 ] selected by the user. This is illustrated in FIG. 26 .
- any actor who receives or sends a determined (absolute or relative) amount of instruction (of either or both types, but generally one or the other as requested by the user) will automatically have a ring of the appropriate line color(s) [ 2705 , 2710 ] or other visual adornment drawn around them.
- both rings will draw, one inside the other.
- the action (sending or receiving) which is more prevalent will be the external ring.
- a user may select the amount of instruction that triggers this display.
- Many embodiments will also distinguish, or allow the user to distinguish, between implicit and explicit instructions by rendering them in different colors [ 2510 ], and/or setting separate visualization thresholds for each. Some embodiments will combine both sender/receiver information and instruction type, thus requiring 4 colors or styles of rings or other visual adornment. Alternatively, the user may choose any one or more of these aspects to display at one time.
- the rendering of rings or other visual highlighting mechanisms is to make the communication patterns more visible in a thumbnail-size image, which in turn allows the user to visually compare many images that represent different time slices at once.
- actor proximity is defined as the measurement of the closeness of social and/or professional relationship of two or more people. In one embodiment, proximity is measured according to the time and frequency of communication, the percentage of communication that is a discussion rather than a singleton communication, as well as whether or not collaborative editing of content is involved. This definition may be extended to include further analysis of the content of the communications. Specifically, the system increases the weight of communications that contain reference to more than one distinct topic as identified by any topic detection mechanism including but not limited to ontologies or topic clustering.
- the sending or receiving over time of communications that collectively contain a large range of different tones relative to the actor's communications generally makes it clear that there is a close emotional or personal relationship between the actors in question. From a visualization standpoint, divergences of behavior from an actor's established norms are the focus. Therefore it is interesting to compare the tone-topic-timing of the communications between any two actors to all other communications of these actors. In this way, the actor's “true” feelings about a particular event or topic may be separated from what she expressed more publicly, which of course has value independent from analyzing the proximity of the actors.
- Some embodiments therefore provide a report on all divergences, which is to say all occurrences where the tone expressed by an actor in regards to a particular topic or event vary with her proximity to the actor she is addressing.
- the time interval to use is user-specified, while in others it is determined by the system via the use of clustering with time as a vector.
- FIG. 28 depicts one embodiment of the display of a set of actors who share both professional and personal relationships, some with augmented weights due to proximity analysis. Note that different tones are designated in most embodiments with different colors as depicted in the legend [ 2805 ] so as to optimally facilitate the creation of readable scaled down images or “thumbnails,” to in turn facilitate useful side-by-side comparisons of different slices of data.
- the embodiment illustrated in FIG. 28 uses cross hatch marks [ 2810 ] to indicate the degree of proximity. Other embodiments may use different visual indicators including, but not limited to, different colors or fills of lines, different styles of lines, or lines that are adorned with various patterns and shapes in order to indicate the degree of proximity.
- FIG. 29 illustrates one embodiment of a visual comparison of the range of tones that an actor has used over time with different actors.
- Arcs [ 2930 ] connect images which represent different actors.
- the pie images [ 2915 ] indicate the approximate relative proportions of the occurrences of different tones in the communications between the two actors in question. (It is “approximate” since in some instances, the same communication can be tagged with more than one tone.)
- the pie wedges [ 2920 ] are embellished with the emotive icon [ 2905 , 2910 , 2925 ] for each tone that is present.
- a mouseover [ 2935 ] with detailed related information is available.
- FIG. 30 illustrates a graphical query view based on FIG. 29 . In some embodiments, clicking on the graphical selection tool, and selecting some objects in the canvas will cause a transition between the type of view illustrated in FIG. 29 , and that illustrated in FIG. 30 .
- FIG. 31 depicts a chart that illustrates the divergence of tone [ 3110 ] in the communication of an actor when speaking to different actors about the same topic(s) within a specified timeframe.
- the timeframe may be specified by the user, or may be determined by applying clustering or other statistically oriented techniques in order to determine the time intervals of greatest interest to examine given the selected actors.
- each tone selected by the user is assigned along the Y-axis of the chart [ 3115 ] and time is shown along the X-axis. Each selected actor is then indicated by a separate line.
- the number of different communication channels that two actors use to communicate with one another is a good indicator of proximity. For example an actor who is frequently using different types of channels of communication such as personal e-mail address, IM, SMS, cellular phone, home phone, and work phone numbers to communicate with another particular actor is likely to have a close proximity to this actor.
- the number of communication channels can be displayed in a separate communications graph.
- mousing over an icon [ 3210 ] which represents the number of different communication channels used for communication between the two actors will display basic information about each channel [ 3215 ].
- the number is color-coded, so that differences in the number of channels available between users is immediately apparent.
- the user can use a control [ 3205 ] to select whether she wishes to see only those channels that were actually used within the constraints of the query as opposed to all of the different channels that the actors had used in all communications with each other. For example, different phone numbers, email accounts, IM identities and electronic identities, and whether one is using text messaging all add up to indicate the level of closeness that exists. Note, however, that this is not a bi-directional measure, as Joe may have every bit of available contact data for Bob—including personal ones—but that the reverse may not be true. A common scenario in which this would occur is when Bob works for Joe. Otherwise put, actor proximity, especially professional proximity is not necessarily symmetric.
- communication graphs can use discussion metrics as their basis instead, or in addition to, of the flow of individual messages. Variations of this idea exploited in different embodiments include:
- FIG. 33 illustrates one embodiment of a graph based on discussion-related thresholds with a set of related controls.
- the system may divide a thicker line [ 3340 ] in two, in order to be able to simultaneously display both discussion and singleton communication. For example, if only one third of communication initiated ended up generating a discussion, and green were used to designate one-off communications and red those belonging to discussions, the red portion of the line would be a third of the total width of the line [ 3340 ].
- each line may have an icon [ 3325 ] at its midpoint that is a cue to the user that a mouseover with detailed related information is available.
- the number of total communications [ 3335 ] meeting the current display criteria are indicated on each line.
- the system uses different visual indicators including but not limited to line thickness, style, patterned fill, and end shapes in addition, or instead of, line color to visualize different aspects of the communication.
- FIGS. 34 and 35 illustrate related embodiments of such an overlay showing discussions. Other embodiments may depict individual messages. It also is often important to visualize the boundaries [ 3505 ] that exist within sub-organizations or between organizations. For example, in large brokerage houses, there may be groups within the organization that are forbidden for some period of time from sharing information on specific topics. Detecting violations of these policies is, therefore, of great interest. In some embodiments, content that is determined to be of a personal nature will be removed from consideration in this view unless otherwise instructed by the user.
- such boundaries [ 3505 ] may be marked with brightly colored and highly visible lines, or other indicators.
- the colors used to designate the organization chart differ on opposing sides of such a boundary [ 3505 ].
- these boundary areas [ 3505 ] are automatically rendered with a spider-eye or wide-widget effect, so as to make relatively larger and more noticeable (relevant) communications that are crossing the boundary [ 3505 ] as illustrated in FIG. 35 .
- all communication is treated in this fashion—not just communication that is deemed to be relevant.
- the user may select the topic(s) that she wishes to see displayed in the graph.
- the user may create her own label for the graph so as to facilitate the use of the graphs as trial art.
- controls [ 3405 ] are provided which allow the user to change certain parameters of the graph, which include but are not limited to: threshold number of messages or discussions, tones, and topics involved.
- the “decorated” view of the organization chart is placed next to the plain one rather than being overlaid.
- FIG. 34 depicts one instance of communication “around” the organization chart [ 3415 ], and one “missing” link [ 3410 ]—a link connecting a pair of actors organizationally which is associated with an abnormally small amount of communication relative to other similarly placed actors in the organization. For example, if a Vice President has six people reporting to him, and there is substantially more communication—either generally, or with specific characteristics such as certain topical content—with the first five as opposed to the sixth, this would qualify as unusual, unless the sixth actor had only been present for a fraction of the relevant time.
- a “heavy” link a link depicting a pair of actors organizationally which are associated an abnormally large amount of communication is also highlighted.
- communication “around” or “outside of” the organization chart is highlighted with a “heavy” link or other differentiated visualization.
- the user determines whether all such communication should be so highlighted, or specify parameters for what should be visualized, such as a threshold for the number of communications, specific properties required of the communications, or limit it to specific organizations.
- the “missing” link is depicted with the red zigzag line [ 3410 ].
- the highlighting may be with an eye-catching zigzagged or jagged line in a bright color, a blinking effect, or another representation used to draw attention to the abnormal behavior.
- the purpose of such techniques is to highlight anomalies that might otherwise be very difficult to spot in a very large and intertwined organization chart.
- this chart may be displayed for overall communications, for a particular segment of time, for a particular topic, or constrained in other ways as defined by the user.
- the system identifies connections which exist in the organization chart but which would not be rendered according to the requirements of the current communication graph (for example, which are below certain user-defined thresholds for communication to be sufficient to be visualized.)
- FIG. 34 illustrates actor-communications, in one embodiment the communication of a group of actors may be displayed collectively by organizational structure or other characteristic.
- One aspect of the system is the use of ontologies and named entity extraction (i.e., the identification of proper noun phrases or acronyms in the text), in order to perform a number of different functions such as the identification of topical content.
- “new” named entities may be identified by analyzing the custom dictionaries, address book information, auto-correction settings, and other types of metadata that may be retained by certain applications, such as MICROSOFT OUTLOOK, MICROSOFT WORD, and trying to assess which entries correspond to named entities.
- new we mean that may not exist elsewhere in the corpus to be extracted. This assessment may utilize a variety of data and techniques, including but not limited to:
- each communication that includes a reference to the named entity is depicted as a link; all other communications are filtered out.
- the color of the link is associated with the timeframe in which the communication occurred. In one embodiment, this may be utilized to determine how the information spread—i.e.
- the representations of actors connected by colored links also take on the same color, either as fill or border, if an image.
- different colors are used to designate the certain awareness of different actors of the named entity based on the interval of time that they first generated content involving the named entity (whether or not the content was shared with others.)
- a date extracted for metadata such as the date when a contact was entered in an address book, will be considered evidence of awareness.
- the links and nodes are rendered against an x-axis representing time; in some of these embodiments, different colors or icons may be used to indicate the medium of communication; for example, email vs. IM.
- the time slice presentation technique described in this application may be used.
- VCR-like controls are provided along with slide sorter metaphor such as those used in common slide presentation tools. Still other embodiments will make use of thumbnail image-based representations.
- While communication style arc-node graphs have many excellent applications, they are not optimal for every use case. For example, they are not particular good at helping to visualize certain types of differences in the properties of the discussion such as when in the discussion some type of meeting event occurred, or if the discussion were abnormally short relative to the normal size of discussions for the particular pairing of topic and actors. Therefore, in most embodiments, the system provides alternate visualizations of discussions.
- FIG. 37 One style of non-communication-graph visualization of discussions that contain meeting object content is shown in FIG. 37 . Note that different visualizations may divide these meeting concepts somewhat differently. For example, the embodiment shown in FIG. 37 shows all calendar events as the same color, regardless of whether the calendar event was referring to a phone meeting or an in-person one (unless there is also a phone record present, in which case the calendar event is a subobject of the actual phone call.)
- each discussion meeting the criteria specified by the user via the query controls [ 3705 ] is displayed as a horizontal bar [ 3715 ].
- the different mediums associated with the sequential items in the discussion are displayed with different coloration and/or icons [ 3735 , 3725 ].
- discussions which end (or “truncate”) in a meeting event are rendered with a thicker border.
- Meeting objects that do have associated text documents are decorated with an icon [ 3740 ] to indicate this fact.
- the system allows the user to query for “mixed type” discussions, that is, those which contain—or are presumed to contain—meetings for which no written record may exist.
- the query language contains operators, and the user interface corresponding controls which allow the user to query for various properties of such discussions. These include, but are not limited to: discussions which end with a meeting object, discussions that appear to contain a pivotal meeting event (discussed in a following paragraph), and discussions which contain meeting objects of specific types.
- Such “mixed” content discussions are by definition instances in which the participants have chosen—at least for some part of the discourse—to move away from written forms of communication. While many of these may be innocent, calculating and noting changes in behavior around key events in the matter (as specified by the user, or as derived by the system,) or simply over time, can be very suggestive. Additionally, to focus analysis or improve visualization, the user may further limit the set of discussions of interest by any number of other means, including but not limited to: the presence of specific topics, as determined by ontological or statistical means, the presence of specific languages, and any method for determining tone.
- requests for a phone call is of a very specific kind in order to be considered an actual request. For example, they must not be conditional in nature (e.g., “if you have any questions, please feel free to call me.”) nor a business formality or pleasantry (e.g., “you may always call me if needed.”) Rather, in most embodiments, such references are limited to imperative forms (e.g., “call me to discuss.” or “let's discuss further off-line”), Otherwise put, most embodiments require that the author be clearly instructing or requesting the recipient(s) of the message to call, rather than merely suggesting that they are there to answer questions or the like.
- FIG. 38 illustrates one embodiment of displaying these probabilities.
- FIG. 38 uses shaded bars to depict the probability that different clusters of discussions (that is, discussions that share a certain number of attributes in common) that meet the query criteria will contain a meeting event.
- FIG. 39 displays comparative data of discussions of different properties as selected by the user with the available query controls.
- This particular example depicts the average length and the total number of discussions for each topic selected by the user, as well as the number that ended in a meeting or phone call. (This last is sometimes referred to as a meeting event or a “call me” truncation.)
- a meeting event This last is sometimes referred to as a “call me” truncation.
- the average size of discussions amongst a specific set of actors involving a customer contract is eight, but for a particular contract it is never more than three, with the last message generally containing an imperative form of “call me.”, otherwise put, a “call me” truncation. This would be a red flag that would suggest that there was something unusual with respect to the handling of this particular contract.
- each discussion that corresponds to the user-specified query is drawn in sequential chronological order that is generally faithful to a timeline.
- each item in the discussion is colored by media type, for example, email vs. IM vs. phone call.
- media type for example, email vs. IM vs. phone call.
- some embodiments may represent each media type on a different line rather than color-coding different types of objects. This is more appropriate in situations where there could be concurrent events that require depiction, for example, IMs sent during a meeting.
- Some embodiments may attempt to identify phone calls as pivotal events based on pragmatic tagging of any written messages that occurred both just before, and just after, the phone call.
- FIG. 40 depicts an instance of this; meeting events that the system has tagged as pivotal are decorated with an icon [ 4010 ].
- discussions that the system deems do contain pivotal meeting events and which lack supporting transcript documents are especially flagged for the user's attention. In some embodiments this is done through additional visual highlighting of the discussion or annotation with an icon. Pivotal events and pragmatic tagging are described in more detail in the SOCIOLOGICAL DATA MINING APPLICATION.
- items that were created on a small format device such as a PDA are identified by the system.
- footers attached by such devices, or the actual statements within the content may be used to identify such devices.
- such items are considered to be a different modality compared to email or IM, even if the communications medium is email or IM. Since most people have difficulty typing on the small keyboard, such items should not be assessed for tonal content on the basis of shortened sentence structure, since doing so would yield false positive results for certain types of tonal content or for “mere forward” analysis. Similarly, a reply to a message sent from such a device must be deemed less likely to be a complete response to a longer message (or a message with attachments), since scrolling to read a large amount of content on such devices is often cumbersome. Some embodiments may consider this dimension when, for example, determining when (or whether) an actor became aware of a particular thing.
- Ontologies and other approaches may be utilized in order to detect the tone of a message or document.
- Other approaches include, but are not limited to: the use of tropes, lexical selection, emotional syntax, analysis of syntactic construction, analysis of contrasting conceptual structure, use of deception theory, and other techniques from computational linguistics. For example, exclamation points and curse words can generally be taken as an indicator of anger or frustration on the part of the author.
- Other tones include, but are not limited to, confusion, concern, sarcasm, etc. This has obvious utility in determining when actors are under emotional stress relative to certain people, or certain topics, which may have a direct bearing on the investigation or litigation.
- each such occurrence of a specific tone is automatically correlated separately and jointly with:
- the visual displays may include Venn diagram visualizations, line graphs, and tabular reports of statistically significant correlations.
- “emoticon” style iconic indicators are present in discussion and item headers and in different display views which indicate use of emotional tone.
- different styles of icon may be used to represent different tones.
- the user also may query for content on the basis of tone or determine that the presence of emotional content either in general or of particular types including but not limited to angry, happy, or frustrated should be considered a criterion to consider the document “hot” for example (which in the parlance of attorneys signifies that an item is both important and interesting), and/or place such items in a special stage in workflow.
- Some embodiments make use of the system's ability to associate quoted text with the actor that originated it.
- One embodiment of creating such an association is described in the SOCIOLOGICAL DATA MINING APPLICATION.
- the association is used to assess not just that there was “angry” language content in the document, but that actor “Joe Smith” was the angry speaker.
- the query language is extended to support queries such as “find all documents in which Joe Smith used angry language.”
- a hostile email from Jane Smith to Bob Jones may simply be indicative of the fact that Jane is having a bad day, or it may indicate that Jane is angry at Bob specifically, or about a specific topic that relates to Bob—or it may be part of a pattern of many people being angry at Bob. It could be the case that there is something going on at Janet's and Bob's company that is causing some group of people to be generally unhappy, and as a result, the steam is being let off that may be directed at someone almost randomly.
- the communications of user-selected actors are analyzed for the presence of different kinds of tonal content.
- Different types of tonal content may include, but are not limited to: angry, depressed, upset, sarcastic, defensive, concerned, confused, happy, friendly, and threatening.
- Each individual item is tagged with whatever tones are determined to exist in it.
- the results are correlated and clustered by other key variables including but not limited to: other actors, time (both calendar, and in relationship to specific events of interest) and topic (as analyzed by any topic detection method). This allows the system to identify the pattern of tonal usage on the part of each individual actor with respect to other actors with whom she interacts, and whether deviations from their own particular norms seem to be triggered by particular other actors, topics, events, etc.
- FIG. 41 depicts one exemplary visualization of tonal content used in communication among user-selected actors.
- the user may select specific actors, or groups of actors, or, by specifying none, indicate that all actors are to be analyzed.
- the user may specify actor-relative groups of actors, for example “Bob Jones and all others in same department.”
- actor-relative groups of actors for example “Bob Jones and all others in same department.”
- data for all actors in the group is combined.
- the user also may narrow the results by topic, time, or specific tones.
- the user also may specify the granularity of the time axis.
- each tone is displayed on the vertical axis of a matrix. As shown, each tone is labeled with its own emotive icon. The name of each selected actor appears as a label once per time increment as labeled on the horizontal axis. For each tone, for each actor, and for each time increment, there is in one embodiment a rectangular area that may be filled in different ways. In the embodiment presented in FIG. 41 , the degree of saturation of the fill indicates the extent of material content of that tone present. Different embodiments may measure this in different ways; for example, as the percentage of all qualifying items that contain that tone, or relative to the normal behavior for that actor.
- a different fill is used so as to highlight the fact.
- Some embodiments also may denote specific co-workers (members of the same organization, however configured in the specific instance).
- the different tones are displayed against the backdrop of a color spectrum, such that, for example, an “angry” tone would have a reddish background and its fill would likewise be red, while a “happy” tone might be depicted as violet in hue, as it is the opposite end of the spectrum.
- the fill may be adjusted based on the norm for interactions between those users. That is, an angry interaction between two users who always interact in an angry tone is less emphasized than an angry interaction between two users whose interactions are normally neutral or happy.
- FIG. 42 depicts a somewhat different embodiment with respect to the visualization component.
- the presentation matrix is the same, but the area of each block that represents the intersection of actor and time increment is subdivided among all of the actors (or actor groups) currently being visualized. If the actor has some of the appropriate kind of tonal content in that time increment, a block [ 4205 ] representing that content will appear.
- the relative size of the block [ 4205 ] is determined by the relative amount of content that exists for that tone.
- the color of the block [ 4205 ] corresponds to the color used to depict that particular actor in the legend. Note that events [ 4210 ] may also be displayed in this view. Note that the exact query controls available vary by embodiment and configuration.
- the example pictured in FIG. 42 is comparing the tonal communication behavior of an individual actor to two groups of actors. Some embodiments will also go the reverse direction, which is to say, given one or more actors, identify actors or groups of actors who share like patterns of tonal usage activity with the selected actors.
- quotation marks around a small number of words are considered to be an indication of both tone and social proximity.
- any word is capitalized that is not apparently the name of a person (as determined by named entity techniques, look-up tables of common names or any other established method) or another known term that requires capitalization, the text in quotes is presumed to be the title of something.
- the maximum number of words is user-configurable. The reason that the number of words is limited is that if the number of words appearing in quotes is small, it is much less likely to be an instance of the author actually quoting someone or something else. A flowchart of one embodiment of this analysis is illustrated in FIG. 43
- FIG. 44 visually indicates quote marks [ 4410 ] if there were one or more instances of presumed “sarcastic” use of quotes. Other embodiments may provide the number of such instances or scale the size of the quotes depending on the number of instances. In some embodiments a mouseover may be provided which displays some basic information [ 4420 ].
- all communications that are to the actor, from the actor, that were created or modified by the actor, or which were about the actor, are considered to be associated with that actor.
- the user may determine which of these associations she wishes to invoke [ 4510 ].
- a stacked bar representation [ 4505 ] that utilizes a different color for each tone such that, for example, an “angry” tone would have a reddish background would be used.
- the bar illustration also may indicate number and percentage of each tone within the communications. In some embodiments the bars will display number count and percentage information upon mouseover as illustrated in FIG. 45 .
- snapshots over time of a meter visualization [ 4605 ] is used, with a dial that can be pointed anywhere from red to violet, depending on the amount of “negative” communication directed at an actor, relative to either all actors, or a user-designated group of actors.
- the user can filter whether all actors, or only some actors are to be considered as input.
- the user may decide to restrict the items being evaluated to, for example, only items received by the selected actor(s), or which referred to the actor, or which is in connection with a particular event or topic.
- each restricted category is assigned its own row.
- the meter visualization [ 4605 ] can also be used with a thumbnail style display or a time elapsed presentation. Key events may be depicted in any of these views with indicators. In one embodiment, the indicators may be shown in the same manner as previously described. Note that some embodiments may forego the use of color in the meter [ 4605 ] but color is helpful for effective use in presentation methods which reduce the image size.
- Some embodiments will visualize clusters of emotive content activity over time, while, in some embodiments, will suppress smaller amounts of such activity for purposes of visual clarity. An example of this is illustrated in FIG. 47 .
- All of the metrics described herein are, in one embodiment, baselined to provide the greatest value and accuracy in determining anomalies. For example, if the period of time of interest for an investigation is the calendar year 2005, it is useful to measure the years 2004 and 2006, to provide a baseline for comparison. While the passage of time will cause some changes in the baseline; for example, an actor may change departments, such differences usually can be easily accounted for, and except in situations of extreme upheaval, are very likely to be the anomaly. Baselines also can be performed on the basis of job title or function as well as on the basis of the individual person—adjusted on this basis. What is important is to be able to isolate unusual behavior, and unusual behavior almost by definition is behavior that deviates from the norm for the set of people in question. As noted in the SOCIOLOGICAL DATA MINING APPLICATION, normal behavior should at least generally correspond to established workflows and other processes as well as the contemporaneously defined organizational structure.
- the system enables users to visually superimpose data for like periods of time in different years, or for example, to compare data on a quarter by quarter basis, since certain activities always happen at the end of each quarter.
- FIG. 48 illustrates one embodiment of visualizing tonal content.
- tonal content presence is assessed for either individually selected actors or groups of actors. Any acceptable method, or combination of methods for assessing tone may be used.
- the display is a matrix showing years in comparison to each other, to allow for seasonal variations in behavior.
- an actor heartbeat is introduced for the purpose of understanding which dates that actor was active as opposed to inactive, in terms of performing any action(s) for which there is a record.
- evidence of an actor heartbeat can come from a variety of sources, including but not limited to: sending emails, sending IMs, sending SMS messages, making phone calls, logging into a system, creating or editing documents, creating any kind of application record instance, or creating calendar events or attending a meeting.
- the frequency of different types of actions associated with a particular actor are visually represented in chart form, as illustrated in FIG. 49 .
- icons reflective of the item type designated by the line are affixed to the line instead of the image representing the individual actor.
- the lines are completely unadorned.
- statistical significance tests are run in order to detect any anomalies, such as sharp unexplained drops in a particular component of the heartbeat. If such anomalies are detected, in most embodiments, they will be annotated [ 4905 ] explicitly in the visual display. In some embodiments, this is reflected in a report format containing content similar to what is illustrated in FIG. 49 . In some embodiments, the extent and type of the deviation from normal behavior that is annotated in the report is user configurable. Types include but are not limited to sharp drops or increases, and high frequency of increasing and decreasing of activity. Types of statistical tests that may be run include, but are not limited to, the amount of deviation from the median line of each activity type.
- FIG. 50 depicts one embodiment of detecting and visualizing potential tampering with backups of an archival format, such as PSTs, the format used by MICROSOFT OUTLOOK.
- a column (line [ 5005 ]) in the chart represents a visualization of the frequency of mail for a particular month from (non-incremental) sequential backup to backup.
- the system visually segments the line into categories, including, but not limited to, when new emails appear, old emails disappear and when there is no change by the use of different colors [ 5010 ].
- Other embodiments may use different color schemes or indications—including different kinds of patterns or other types of visual representation for additions and deletions other than solid color fill, or allow the user to determine a display differentiation scheme.
- a month's non-incremental backup should have everything that the prior month's backup included, plus new email.
- the system visually indicates [ 5015 ] backups that should be scrutinized further as a result of possible tampering.
- a mouseover [ 5020 ] with detailed related information about each backup is available.
- Some embodiments also will generate a report format listing any usual occurrences, such as this scenario.
- the X-axis depicts calendar time at whatever level of resolution is set by the user, while the Y-axis depicts the individual non-incremental backup files.
- the document has a document lifecycle associated with it.
- This is a set of stages that documents of the given type typically move through, from initial creation to ultimate abandonment.
- the stages may be highly formal, such as an approval entered via a commercial workflow system, or informal such as someone sending their manager a copy of a report they are working on to edit—or even additional sessions of editing the document by its original author. Some stages may be required, while others may occur either not at all, or more than once.
- any email that was saved to the “drafts” or similar folder, and any document that was sent out as a draft can be considered to be a stage in the lifecycle of that document.
- draft documents are identified based on the inclusion of the word “draft” in a title, in a header, or in some embodiments in the first paragraph or page.
- other data formats such as images or faxes, may be deemed drafts, if the associated metadata provides such an indication.
- a reference to the document being a draft in the parent email of an attachment also is deemed sufficient. This is because the notion of a “draft” suggests a multiple step process of review that is often associated with more important documents.
- each sending out, checking in, publishing, or posting of a document is considered a lifecycle event, whether or not it corresponds to an ad hoc or structured workflow that might exist for that type of document.
- the relationship of the recipient(s) to the sender may modify the type of lifecycle event, and whether or not it is considered a major or minor one.
- sending a document to a subordinate in one embodiment is a different event from sending to a peer, to a supervisor, or to higher organizational levels—or out of the company.
- either or both the number, and number of different types of recipients also may impact the category of the lifecycle event. In many embodiments, this behavior is user-configurable.
- burst of activity as a bounded period of time in which a large number of edits are made to a document after its initial creation.
- bursts of activity are colored or otherwise indicated so that they can be more easily detected when the view is zoomed out.
- Documents may start with an official “draft” stage as above, or may simply go through a large number of modifications or edits in a relatively short, or at least bounded period of time, may be get sent to several people, may be edited several times again, then may sit dormant for a spell, and so on.
- Different embodiments may use data clustering algorithms (including agglomerative techniques, since bursts of activity may be hierarchical) and other similar techniques in order to assign to the bursts of activity, in some embodiments, both major and minor version numbers. These are used both for general reference, and for labeling of the burst or other editing activity in the document lifecycle view.
- major numbers are assigned by burst of activity
- minor numbers by sequential non-auto-saved (that is, only those changes explicitly saved by the user, rather than those which are automatically saved at short intervals by the application) changes. Note that these minor numbered changes may fall within a burst of activity.
- every change is sequentially minor versioned.
- FIG. 53 A flowchart illustrating one embodiment of versioning is depicted in FIG. 53 .
- Singleton changes that is, those that are not substantive enough to qualify as bursts—may be given a major number or continue as a minor number from the last burst of activity, depending on the embodiment.
- FIG. 51 illustrates an example of one embodiment of a document lifecycle view in which two document lifecycles are depicted along with a related periodic event, in this example a board meeting [ 5105 ].
- the lifecycle view displays changes to a document which include but are not limited to creation, modification, revision, deletion, check in of a revised version, check out of a revised version, edited as draft, sent as draft, or mentioned as a draft, reviewed, approved, or rejected, as may be seen in the legend [ 5110 ].
- mouseover information including but not limited to the actor responsible for the lifecycle event, the exact time and date, and other detailed information about the given lifecycle event will be displayed.
- what gets displayed is user-configurable.
- multiple document lifecycles can be displayed in a single view. Most embodiments provide different options for doing this, including but not limited to:
- workflows are considered to be just another type of document lifecycle. Some embodiments may visually designate workflow-related document lifecycle stages separately. In some embodiments, and as will be pictured in FIG. 52 , the time interval between two workflow-related lifecycle events may be labeled with a line with arrows on both ends, and which indicates its length at the midpoint of the arrow [ 5210 ].
- each document lifecycle event is depicted with a small icon [ 5130 ], the color of which is determined by the type of lifecycle event the icon is representing.
- a legend [ 5110 ] provides a mapping of available lifecycle event types, colors, and/or fill types that are being used to represent them.
- the icon's [ 5130 ] position on the X axis indicates when the lifecycle event it is representing occurred. If more than one document lifecycle is being displayed in the same view, in some embodiments, each lifecycle is assigned a horizontal slab of the view to occupy. In other embodiments, each icon [ 5130 ] is numbered to indicate which document instance it is associated with. Some embodiments may do both, and/or use other techniques to indicate association. In some embodiments, those icons [ 5130 ] which represent portions of bursts of activity will have the background area around them darkened. This is to help make the view more readable when shrunken to thumbnail size.
- an extensive set of manipulation tools are provided, including but not limited to: panning widget, spider-eye panning widget, zoom, find in canvas, and drawing tools so that the user can annotate and save the view.
- the system in one embodiment, has the ability to line up ad hoc—that is, empirically observable—and/or structured workflows with the same or similar types of documents within the same window so as to more quickly detect anomalies in behavior.
- ad hoc that is, empirically observable
- structured workflows with the same or similar types of documents within the same window so as to more quickly detect anomalies in behavior.
- FIG. 52 A separate view [ 5205 ] which depicts an abstract workflow has been added above the one displaying different document lifecycle instances [ 5210 ].
- the abstract definition of the workflow may be extracted from a workflow system. We call this a structured workflow. Alternately, if provided a sufficiently large number of instances of a given type of document content, the system may adduce an ad hoc workflow as described in The Sociological Data Mining Application. As shown in FIG.
- each step in the workflow is titled [ 5215 .]
- the title of each step is part of the information extracted from the workflow system.
- the system will attempt to assign a title in some embodiments, while others will simply not provide one.
- mousing over the title will provide information about the step, including but not limited to who is responsible for the step, and the expected or required completion time for the step. Note that in most embodiments, the abstract workflow view [ 5205 ] will only show steps that are required by the workflow.
- Automated dynamic token updating such as a date field automatically updating when someone opens the document.
- spelling corrections may be evidenced by a word that was not in the standard dictionary or known list of specific terms, such as actor names, being replaced with one that is.
- Other embodiments may exclude changes to header or footer content, changes to dynamic updating token behavior, addition of new records to existing ones of the same type already in the document, and other generally ancillary changes.
- changes that are made in two editing sessions or less (where in one embodiment, a session is defined by opening and closing the document, and in another embodiment it is determined in other ways, such as bursts of activity) and which all changes are contiguous AND which are not proximate to any event of interest.
- An event of interest could include a specific event defined by the user or an event type. In one embodiment, all of these parameters are user configurable. Proximity to an event of interest can be critically important insofar as even a minor change, such as hiding a column in Excel may take on a much greater significance if it was done just prior to the sending out of the document as part of a board of directors' information packet.
- Bursts of activity can also be queried as a property of a document instance, family of document versions, or of actors. Reports that correlate bursts of activity with events, as broken down by different document types, documents with specific types of content, and by actor/actor group are provided in some embodiments.
- the system may create inferences for the user based on such historical correlations that the presence of a particular burst of activity suggests the proximate occurrence of a related key event that the system currently does not have knowledge of.
- An “Item Information Report” is to capture the detailed history of events for that item. In some embodiments, this is further extended to include related items, such as ancestors and descendants.
- Chronologically sequential list of all dates and times that are in any way associated with the item include, but are not limited to:
- List of actors who sent and who received the document in some embodiments sorted by organization, to help highlight actors outside the authors' department or company who received the item.
- Log of comments made by different actors about the item including but not limited to: comments that are extracted from check in logs of document management systems, forwarded content in emails, and “user comments” entered in a text editing application amongst other things.
- the Item Information Report is a more detailed, textual companion to the Document Lifecycle View, from which it may be directly accessed—in some embodiments by clicking on a button. In some embodiments, it may also be accessed by clicking or double clicking on the individual document icons or on the appropriate item title at the top of the view. In some embodiments, the user also may access the report directly from an icon in the item header in the discussion transcript view.
- Spreadsheets and other fielded or structured documents are particularly difficult to apply standard information retrieval techniques to, because much of their content is numerical, or consist of common words such as the names of different months.
- a small change to a formula that affects a large number of cells may well make two versions of the same spreadsheet that look quite dissimilar, when in fact there is only one small difference, or may obscure the fact that someone is reusing a boilerplate or template spreadsheet. This poses a significant problem when one is trying to determine the ancestral lineage of such documents, as well as assessing what it logically should mean to say that two spreadsheets are “similar.”
- FIG. 55 present a flowchart of one embodiment for identifying these cases of close ancestral relationship for spreadsheets.
- the system extracts all spreadsheet formulas.
- alternative methods may be used, such as any type of exporting of the contents of the spreadsheet.
- the system can determine whether two spreadsheets differ by dint of manually entered information in cells, formulas that govern these cells, a combination of both, or by ancillary characteristics like cell or font formatting.
- the system first checks to see whether either spreadsheet has formulas, and if so whether they are the same, or if one is a superset of the formulas of the other.
- Spreadsheets may be determined to be related to each other as a result of lexical similarity, the same actor's involvement, a clear record of transport between different actors associated with the different spreadsheets, application meta-data as to version or change-tracking content, or other reasons. If the formulas are the same, or supersets, this is treated as positive evidence of a relationship. If the formulas are different—even if similar—the system can identify the number of cells that differ because of governing formulas versus manual entry, by applying the formulas from spreadsheet A to spreadsheet B and vice versa. If the number of non-formulaic changes is small, this is evidence of a version relationship. In some embodiments, the user may determine the threshold for determining a “family match.”
- the export option also allows us to grab other potentially contextually important properties of a spreadsheet or other related document type, such as the hiding of a column, cell or font format information. Once such information is extracted, it may then be compared against other likely versions of the document in order to determine when exactly the changes were effected, or in one embodiment, to ascertain that the change(s) in question occurred between two specific points in time. This can be useful in litigation. For example, hiding a column or changing some visual characteristics so as to de-emphasize certain information right before it went out in an SEC filing or other key event can be used as evidence of fraud.
- the system may include a tool that allows for both the display and editing of the on-line privilege log, which allows queries to be done on any of the fields, and allows both the individual editing and global search and replace of privilege claims or other user-designated fields accordingly. So for example, let us say that Jane Reviewer has consistently inappropriately used the privilege claim of “attorney client communication” instead of “attorney client work product.” An administrative user may globally correct the errors with a single find and replace operation. In this way, a large number of changes can be made with only a few keystrokes and mouse actions.
- FIG. 56 illustrates one possible example of an editable privilege log. As depicted in FIG. 56 , it features a global “find and replace” widget [ 5605 ] and provides a tabular display [ 5610 ] of all documents that are currently designated as privileged. Different embodiments may display somewhat different controls and data for each document. In the example provided here, controls are present to download the item [ 5620 ], and to view the document by clicking on a link [ 5615 ]. Most embodiments also feature print and sort buttons. In most embodiments, when the privilege log is printed, it is printed without any of the controls. In some embodiments, the privilege log itself can be downloaded in addition to the individual items.
- Some embodiments may also provide user interface functionality that allows the user to input rules into the system for automated assignment of privilege claims, for example based on actor, topic, or medium.
- the user may use the editable privilege log or bring up the item in any view offered by the system that provides coding controls in order to correct any errors made by the system.
- the present system may provide a visualization of all clusters that were used for content categorization purposes, in which each cluster [ 5705 ] is proportionally colored by what percentage of the items in it were found to be responsive, non-responsive, or privileged by human reviewers.
- FIG. 57 illustrates a common logical partitioning of the decision space for the litigation use case.
- the interaction in one embodiment, is divided into colors as if it were another cluster. This visualization is highly useful in that it helps pinpoint those clusters which will likely require individual review of each document in order, for example, to completely comply with a subpoena.
- graphical querying is supported in this view.
- text content that was successfully extracted from the messages in question also can be pairwise compared, specifically excluding consideration of the email agent artifacts, which can include but are not limited to: common strings that can be matched with regular expressions, such as “At 5:50 AM, Jim Quick wrote:” and non-standard alphanumeric characters, such as various quoted text markers that are email reader dependent. This is critically important, since such differences will invariably change the hash of the content.
- this content comparison is done by creating a frequency table or bag of words for each item and diffing them. In one embodiment, this is done for each individual text block so as to make it more resilient in the face of larger amounts of corruption.
- the content is compared by contiguous text blocks to determine whether or not it actually is a hash collision, or rather a case in which one of the items has become corrupted in some fashion.
- the system attempts to see if any textblock in the first email matches any in the second. If this is not the case, either one copy is completely corrupted, or there is in fact a hash collision. In this scenario, the latter has the far greater likelihood of being true.
- “words” that are not well known or acceptable strings in the context of the current encoding and language scheme are noted and presumed to be evidence of corruption. This is logically illustrated in FIG. 59 .
- all emails with identical header-content-based hashes will have this content comparison done as a precautionary measure. In other embodiments, it will only be done when there is an explicit reason to suspect corruption, for example certain types of system exceptions thrown while endeavoring to parse the content.
- the user interface clearly displays a warning message and or icon [ 6005 ] highlighted in a bright color to indicate that a substitution has been performed. An example of this is illustrated in FIG. 60 .
- substitution might not be desirable. For example, if a lawyer wants to claim that lie was unable to view a document because it was corrupted, it is important to note that a substitution was made. The user can then choose whether the good copy should be substituted for the bad when producing documents.
- the system in one embodiment provides a screen which utilizes a metric for determining the “distance” between two pieces of text in order to identify names that are suspiciously similar, and hence have a high likelihood of actually being the same person.
- the metric in some embodiments is a well known metric such as Levenshtein edit distance, also know as just “edit distance.”
- Levenshtein edit distance between two strings of text, the source string s and the target string t is the minimum number of deletions, insertions, or substitutions required to transform s into t.
- Other embodiments may utilize a different metric instead, or employ multiple metrics.
- FIGS. 61 and 62 illustrate one embodiment of the user interface which may be used with this method.
- the user interface displays the files and folders of the custodial data in a tree structure as illustrated by FIG. 61 . It allows the user to choose a custodian from a pre-existing list, associate a custodian with each folder or file, or add a new custodian, through the GUI.
- the system sorts the custodians from which the user chooses edit in order by the “distance” from custodian's name to the name of the particular file or folder. For example, if the name of the folder is “kschon”; the custodian “Keith Schon” will be listed before “Roman Brouk” or “Joanes Espanol”.
- the distance between the name of a file or folder and the chosen custodian may be indicated visually.
- the different degrees of matches [ 6115 ] are indicated by color—red, orange, yellow and green to illustrate the worst match to the best match.
- different embodiments could use other color or visualization schemes.
- the different levels of matches [ 6205 - 6215 ] between custodian name and the name of the file or folder is illustrated in FIG. 62 .
- the “best guess” button [ 6225 ] will populate the custodian slot with the closest name for every selected file or folder.
- the “exact file or folder name” button [ 6220 ] will only populate the custodian slot with a name exactly matching the file or folder name for every selected file or folder.
- One embodiment of identifying actors algorithmically assigns electronic identities to human individuals. Unfortunately, no algorithmic method (nor human method) can accomplish this task perfectly. Often there is no good way to establish the real identity of bluebear@yahoo.com for example, (other than being directly told in a witness interview). At other times, two actors with very similar names and behavior patterns may get understandably confused with one another. This is highly problematic in litigation, when the actor(s) involved with a document may determine whether or not it is responsive to the subpoena.
- the system provides a user interface whereby a user may correct actor information based on their independent knowledge of the real world facts.
- the user may do any of the following tasks illustrated by FIG. 63 :
- an electronic identity can include, but is not limited to: email aliases or addresses, IM handles, phone numbers.
- Attributes include, but are not limited to: privileged status, organizational membership, and country of residence.
- the system automatically presents lists of similar actor names that are associated with different identities, facilitating the identification of possible issues, verification of correct mappings, and where needed, correction.
- Some embodiments may use Levenshtein edit distance as was discussed in a prior section on correcting custodial information. For example, in a large enough corpus, one might have many J. Smiths, and even several “John Smith”'s. Both human reviewers and algorithmic approaches may understandably confound such similarly named actors.
- any user actions performed through this user interface are automatically sanity checked before the new information is integrated into the system. This is to prevent a case of user error doing substantial harm to the system—and a resulting failure to respond properly to a subpoena if this mapping is done in a litigation context.
- Sanity checks may include, but are not limited to, one or more of the following:
- a privileged actor attribute is changed by the user, the system checks to see if the change is generally consistent with human reviewer decisions involving this actor. Specifically, if documents authored by this actor were not generally reviewed by reviewers as being privileged in nature, in most embodiments, the system will provide a warning message, since presumably if the actor were in fact a privileged one, one or more of the reviewers would have realized this. In some embodiments, a threshold number of such conflicting reviewer decisions could be set by the user to determine when a warning message should appear.
- a linguistic fingerprinting check is done to verify the correctness of the request. More simplistic methods may also be utilized by some embodiments. For example, if there were a request to reassign an identity from actor “Bob Jones” to actor “Roberta Jones,” but the emails of this identity were all signed “Bob”, this would raise a red flag. (This can easily be accomplished with a lookup table which maps common nicknames to full names.)
- a sanity check fails, in many embodiments an alert is sent to one or more administrative users of the system.
- the change will not be processed unless an administrative user verifies it.
- the as-yet-unapproved change may be indicated as pending during this interim period. Once approved, the changes may include, but not be limited to:
- a simple user interface allows the user to specify highlighting scheme by:
- FIG. 64 One embodiment of this is illustrated in FIG. 64 .
- the system provides a special query screen that focuses on people—both system users and actors from the corpus—and the actions that they may have performed with respect to specific items in the corpus.
- this query screen offers a joint pick list which combines both system users and actors from the corpus, a picklist with a set of actions, and a set of options for content selection.
- Both pick lists include an all-inclusive entry; “anyone” in the case of the system users & actors pick list and “all” in the case of the “action” one. This allows the user to query for the full list of people—whatever their role—who have seen X or Y.
- the set of actions/choices includes, but is not limited to:
- the set of content options includes, but is not limited to:
- Custom query e.g. “any spreadsheet document created by Actor A that includes the word “turtle.”
- the results are displayed in a user-configurable view.
- the data is presented in a tabular view format, which allows the user to see for each retrieved item all review-related actions as well as which actors operated upon the item in some way.
- Some embodiments may support direct use of a query language or other more sophisticated mechanism so as to allow the user to specify different actions associated with different persons in the same query.
- a tally of review decisions for both individual items and the group of items returned is also provided.
- the user may perform a query to see all near-duplicate items for a particular item, and this may result in 100 items being retrieved.
- the 100 items may have been reviewed a collective total of 250 times (since it is generally true that more sensitive documents are reviewed more than once).
- the user interface in one embodiment, will chart how many times these items were found to be responsive vs. privileged vs. non-responsive or how they were categorized with regard to any of the different categorization schemes.
- the system may further chart how often the documents were found to be in other categories, such as “hot.” This reporting helps identify review consistency problems that are related to specific families of items.
- One of the important use cases for this functionality is to identify any situations in which the system user is also an actor with respect to the corpus that is being reviewed. This is a situation that happens, for example, when outside counsel is reviewing a collection of their client's documents that includes documents and emails which originated at the outside law firm.
- Information from reviewer reports can be used to create metrics to evaluate the quality of the reviewers—the good reviewers are the ones who make the desired decisions consistently in the least amount of time.
- the information derived from processing this data can also be used to assess the effectiveness of specific users reviewing documents with different characteristics.
- characteristics may include, but are not limited to: document type, amount of graphical or fielded content, topical content, length of document, or foreign language content. These characteristics could also include more litigation-oriented designations, such as responsiveness, or associated custodians.
- the system in one embodiment, provides a user interface whereby a designated administrator may create batches of items (or collections of items such as discussions or clusters) that have one, or a combination of N of these properties. For example, a batch may be specified to contain only items that are in French and involve discussion of tax code issues. Having defined batches in whatever way corresponds to both the objectives of the review and the individual strengths of the reviewers, these batches may now be assigned to different individual reviewers as appropriate. This is logically illustrated in FIG. 66 . An exemplary implementation of the user interface for assigning batches is depicted in FIGS. 67 and 68 .
- the system provides the user with two types of controls with which to perform batch assignments. The first of these allows the user to assign individual batches to other users. The second allows the simultaneous assignment of N many batches at once.
- the user is provided with a set of filters that she may use in order to limit the number of batches for which detailed information will be displayed, and which she may assign out.
- a control [ 6805 ] is provided which allows the user to specify how many batches should be assigned at once.
- prior bulk or individual batching assignment commands can be automatically rerun, resulting in the next N batches of the prior type becoming automatically assigned to this user.
- the system makes a determination automatically of what kind of batches should best be assigned to a reviewer based on their track record for speed and accuracy of decisions.
- the present system will use techniques including ontologies and topic clustering in order to determine the presence of specific topical content.
- items and discussions can be initially slotted into different starting workflow areas. This allows items with more relevant content to be prioritized for earlier review, or review by more senior or trusted users.
- automated categorization methods also allows for a sanity check on the human reviewers. Specifically, the sanity check verifies that human reviewers who are all reviewing items that were determined to contain a particular topic of interest, on average, all agree or disagree that this content is present roughly the same amount.
- these metrics are used to project review completion times, based on the current number of documents, the currently available reviewers, optimal batch type assignments in which documents are assigned to the reviewer(s) who can most efficiently review them, and the maintenance of current performance characteristics.
- Some embodiments will perform load balancing calculations in their completion time estimates. For example, they will assume that documents with certain characteristics will be given to somewhat less efficient reviewers if this will result in an earlier completion time because the most efficient reviewer(s) for that document type are already overloaded.
- the user may provide input to the system through user interface controls (not shown) in order to dynamically model the review effort. For example, if the review must be completed by a certain date, how many more reviewers—and with which specific skills—would be needed? Or how many more hours a day must the currently available staff of reviews work in order to meet the desired deadline, if indeed this is still a possibility. Or how much eliminating a stage of review would advance the goal.
- Some embodiments may also allow the input of cost information in order to help the user determine the least expensive course of action as well as the fastest.
- This method will yield much more accurate projections than a projection simply based on an average time per document, since some types of topical content take longer to review than others, and reviewers are likely to review much faster when given content that they feel comfortable with. For example, if we know that Jane Jones is the most efficient at reviewing Spanish documents relating to product defect issues, and we have assigned all such items to her, because we know how long on average it takes her to review each such document, and how many such documents there are, we can fairly accurately project when she will finish.
- the system enables the user to constrain the clustering by the objective constraints of responsiveness including, but not limited to, the date range, the relevant actors or custodians, and then generate a report of instances in which reviewers—or for that matter the categorization mechanism—differed in their decisions on items that were both very statistically similar in content and which are not precluded from being responsive in some way.
- Categories of responsive items are typically associated with one or more specific line items in the subpoena or similar request for information.
- the responsive categories may vary quite extensively, from very specific and detailed questions to broad requests for anything that is relevant to some topic.
- the items identified as privileged are generally withheld from turnover but must be identified, and an inventory of these items, a privilege log, must be presented. Because of the stakes generally involved in a legal matter or investigation, high levels of accuracy and comprehensiveness are necessary. The level of accuracy required is much higher than is generally acceptable for use of standard categorization methods in other domains (for example categorization in web search engine results). Obtaining the necessary performance requires a process that blends the results obtained by applying a potentially large number of disparate categorization techniques.
- the automated production process described here includes an automated production system which generates rules for combining results from a heterogeneous collection of individual categorization techniques and then applies those rules to rapidly produce high quality categorizations of large datasets.
- the system is designed to work with a large number of approaches, including one or more of search engine style queries, general clustering techniques such as classifying items by topic, and custom techniques tailored specifically to a given data set.
- the invention described herein may be applied to any categorization task on large datasets, not just the primary one described in this application.
- FIG. 96 describes an overview of the elements in one embodiment of the automated production process. This process does not address the details of any of these categorization techniques individually, but rather focuses on a systematic method for combining the result sets derived from these techniques into simple container objects that we will call categorization components [ 9637 ], which may then be manipulated as homogeneous objects in order to achieve highly accurate categorization results. This method also makes it more practical to utilize pre-existing “off the shelf” work, for example libraries of generic ontology classes in an ontologically based categorization technique.
- a categorization component is defined to be simply the set of items found to be in a category [ 9638 ], however generated or arrived at. This simple definition provides consistent grounds for comparison between very different techniques, whether a categorization technique such as an ontologically based query where results are assigned to a predetermined category or a technique such as a data clustering approach that discovers an unknown number of categories in the data.
- a categorization technique such as an ontologically based query where results are assigned to a predetermined category or a technique such as a data clustering approach that discovers an unknown number of categories in the data.
- categorization components are not intended to ultimately categorize the data set. Rather, they are an intermediate set of categories that describe items in the data set.
- the method formally defines the description [ 9630 ] of a member of the dataset as the set of categorization components containing it.
- the final category or categories assigned to an item, its production category [ 9624 ] will be a function of its description, called a production category mapping [ 9640 ]. That is, the method constructs a function mapping from descriptions to a final production category.
- production we mean the act of ultimately turning over data to the opposing side and the court. However, the method is in no way limited to that use; the production category can be thought of as the category ultimately assigned through the use of the method.
- categorization components would capture distinct characteristics of an item that could then easily be mapped to a production category based on simple Boolean conditions. For example, in order for an item to be responsive to a subpoena it might require that the item have remediation language in it, refer to one or more of a specific set of companies, and mention certain types of products. With a set of three categorization components cleanly identifying the sets of items with each respective characteristic, the production assignment would simply be an “AND” over those three elements in the description of an item. However characteristics often have more subtle inter-relationships between them. This has the effect of creating correlations between components, i.e. the membership sets have non-empty intersections.
- the system identifies the largest sets of items with independent characteristics in a particular context (i.e. the particular categorization components for which the characteristics are independent). This is balanced with a dual purpose of finding the smallest sets of items for which independent characteristics can't be found. Out of the resulting collection of sets, the system then finds those that are most consistent with reviewer results and generates a production assignment function based on them. This is one step in the overall process, which seeks to refine the categorization component set by identifying and modifying those components that contribute sets of items that are either too heavily correlated with other components or for which there are no useful subsets that are consistent with reviewer results. By modifying components to reduce these problems, a better and faster production assignment rule can be constructed.
- the overall process distinguishes the role of an analyst [ 9627 ], from that of a reviewer [ 9641 ].
- the analyst will manage the overall process and they will directly implement or manage the application of categorization techniques to produce categorization components.
- Reviewers will make categorization decisions [ 9636 ] on review sample sets.
- This process may be combined with additional manual review of items which are deemed particularly sensitive or critical by dint of their content, author, or any other reason.
- one or more of the production categories may be intended for additional manual review. In other embodiments, these items may have been separated out before the automated production process begins.
- a review effort may initially proceed with manual review, and then switch to the automated production.
- randomized samples can initially be drawn from the reviewed material so long as they are drawn in a way that respects a stratification of the corpus, described below. In all cases, however, random samples must be drawn in order to create the supervisor data set.
- stratified sampling when drawing reviewer samples [ 9621 ].
- stratified sampling consists of splitting a dataset into subsets and drawing samples from each independently. The sizes of the samples are not scaled with respect to each other; rather they are determined by the normal statistical significance tests used to determine sample size. When producing any overall statistics then the results from each strata must be weighted to account for the size differences between strata.
- the reviewable universe [ 9631 ] is stratified in order to allow for more focused areas of review. This could be because there are reviewers with different areas of expertise and specialization, or because of special review requirements for certain types of items.
- the consistent element for any project will be to separate out items that are “junk”, in other words of no interest.
- “junk” items are of no interest in the sense that there is really no chance for them to be responsive (for example spam email). Therefore, as an initial step, most embodiments seek to set aside a “remainder” set [ 9626 ] as quickly as possible.
- this set is defined as the complement of the set of items that are believed to have some positive probability of being responsive—or in other applications, of interest to examine. For our purposes, we will call this set the supra responsive set [ 9625 ].
- Most embodiments will deliberately use a coarse, or broad, categorization technique in order to define the possibly responsive set, as it is desirable to direct most of the review effort to those items. In some embodiments this may be done by creating a simple, overly broad, keyword query. The query could be constructed only to remove clearly non-responsive material, or could be constructed under the broadest possible definition of relevance. Other embodiments may use different methods.
- Supra-Responsive Query This initial query, or other equivalent coarse categorization method, will be called the Supra-Responsive Query. No preset level of precision is necessary for this query; the purpose is to minimize the time and effort spent. In some embodiments, a good target for the size of the supra responsive set is about 3-4 times the estimated size of the ultimate responsive set.
- the analyst's initial estimate of the size of the ultimate responsive set may be based on a combination of experience with prior corpuses, and initial surveys of the specific corpus. In one embodiment, the estimate may be refined as the sampling process moves forward.
- the definitions of responsiveness are still fluid. Some parts of the information request may be open to renegotiation and the overall strategy of the case is less settled.
- the process of building up the initial separation into supra responsive and remainder strata provides an orderly method for performing an exploratory survey of the data while the definitions firm up. Additionally, focusing review within the supra responsive set allows for reasonable predictions of the number of items that will ultimately be assigned a responsive production category. Therefore in one embodiment, a sample size can be selected so as to result in a statistically significant number of items with positive responsive calls.
- the related measures of coverage [ 9611 ] and over-generation [ 9612 ], are used for scoring the degree to which a production categorization fits [ 9606 ] reviewed samples.
- Coverage and over-generation are closely related to the precision and recall measures that are very commonly used in the field of information retrieval. Coverage is the percentage of cases for which the assigned production category agrees with the production category mapped from reviewer coding decisions.
- Over-generation is the ratio of total items erroneously assigned to a production category over the number of items found to be in that category, i.e. labeled/reviewed—1. It is the inverse of the precision measure used in information retrieval. In most embodiments, the user specifies the desired relative weighting of coverage versus over-generation in order to meet their specific current needs.
- FIG. 97 describes one embodiment of the decision steps involved in the process.
- discussions are presented to reviewers in order to deal with the problem of correctly categorizing short format items into the proper production category.
- a permutation of the list of all items in the reviewable universe is constructed [ 9705 ]. In some embodiments this may be further restricted to a subset of the reviewable universe.
- the permutation list is formed by randomly shuffling a list of item IDs. In one embodiment, this permutation list is stored in a database that keeps track of all samples drawn during the course of the process. In one embodiment, no item appears in more than one sample. Using a fixed, randomized permutation helps to protect against sampling biases, as parameters and sampling strategies are adjusted over the course of handling a particular corpus.
- the selection of N items from a particular strata is implemented by doing a sequential scan over the permutation list stored in the database and taking first n items that: a) have not been part of a previous sample, b) have not been marked as being excluded from review, and c) are a member of the set of items in the stratum.
- n items that: a) have not been part of a previous sample, b) have not been marked as being excluded from review, and c) are a member of the set of items in the stratum.
- samples from all strata in a stratification of the universe are drawn at the same time, and the entire set of changes to the database are updated atomically in one transaction in order to ensure that a set of samples are drawn from the same stratification.
- samples are selected, in some embodiments they are placed in batches to be assigned to reviewers. Since the order of sample items has already been randomized, in some embodiments batches are assigned by taking consecutive sequences of the desired batch size from the sample list.
- the initial samples are drawn only from the remainder set with instructions to reviewers to identify anything that may possibly be considered responsive [ 9707 ]. In one embodiment, if a significant number of items in the remainder set are marked as possibly responsive by reviewers [ 9709 ] the initial query is modified as appropriate [ 9708 ], and another round of sampling is performed. In one embodiment, this refinement is iterated until a satisfactory level of coverage is established, i.e. it is determined that very few responsive documents appear in the remainder set. At that point the reviewable universe is repartitioned into the Supra-Responsive and Remainder strata. The sampling process then enters a phase where samples are drawn from the Supra-Responsive set exclusively. As noted above, there may be several sampling strata within this set.
- the process is intended to be iterative at two levels.
- an “outer loop” the analyst draws a sample for the primary categorization review [ 9710 ] and assigns batches out for reviewers. If after review the number of items with coding decisions that will be mapped to a responsive category are too low, further samples are drawn until a sufficient number are found [ 9711 ]. Then in the “inner loop” the system initially calculates a production rule assignment [ 9715 ] representing a “best fit” of the current categorization components to the sampling results generated on the current iteration of the outer loop.
- the analyst uses analytic tools provided by the system to determine whether improvements can be made to the current categorization components [ 9719 ]. If no then a new set of categorization components are generated and another iteration of the inner loop starts [ 9718 ]. If at block [ 9719 ] it was determined that there are no useful splits or refinements to the current set of categorization components, then the categorization scheme must be re-examined and categorization components may be dropped, added or recombined with other categorization components [ 9714 ], before re-entering the outer loop.
- the process moves on to a verification step.
- this step allows the legal team to certify results in a turnover.
- the final categorization is computed for the reviewable universe and the universe is then split into two strata, one for categories that are to be turned over and one for categories that are not.
- samples are drawn for verification review [ 9720 ].
- the final categorization is computed for the reviewable universe and the same strata from earlier in the process are used.
- the quality of the final categorization is calculated using the same coverage and over-generation measures as discussed earlier.
- a special coverage number is calculated for the non-produced set only. This coverage is based on what percentage of the sample from the non-produced set is determined to be non-responsive. In any case, if performance goals are not met, the process re-enters the inner loop at block [ 9715 ].
- the final sample drawn for verification of coverage and over-generation numbers is also stratified, but the split is between the produced and non-produced subsets of the corpus.
- non-produced subsets of the corpus include non-responsive as well as privileged data.
- the final sample is reviewed blindly.
- items from the two strata are mixed randomly in reviewer batches with no indication in the user interface as to which strata an item belongs.
- sample sizes are determined by a statistical significance test.
- a size is chosen so as to have an expectation of producing a large enough number of positively reviewed items in order to achieve statistical significance. In the initial round or two, the number achieved may be off because the responsive rate is simply higher or lower than expected.
- the percentage of positive responses—i.e. produce or privileged—is high enough it is in some embodiments advisable to go back to the first step and rebuild the supra responsive query. In most embodiments, it is a reasonable expectation that there might be 4 or 5 rounds of review of the results of the Supra Responsive Query, and a similar number of rounds of review of the more detailed categorization components, plus the final verification round.
- the options presented to reviewers in the user interface for document categorization purposes may not correspond directly to production categories; in those embodiments the production category is deduced or mapped from the coding decisions. In most embodiments, any mapping of the user's choosing may be used, so long as it maps each categorization option in the user interface to a production category. Therefore we will distinguish between categories used to produce categorization components, final production categories, and user interface categorizations. Some embodiments only offer reviewers a choice of responsive or non-responsive; in many of these embodiments, privileged items are handled in a separate, subsequent step. However other embodiments may provide additional options to reviewers.
- a user interface categorization such as “arguable” may be available.
- the “arguable” category can be presented to reviewers as an option for items that are in a gray area, or about which they are uncertain for some reason, or for which there are still unresolved issues.
- a state of “arguable” may be assigned automatically by the system when an item is reviewed by more than one reviewer and the reviewers disagree. In some embodiments these items are reported for remediation and/or assigned to a more senior reviewer to adjudicate.
- the three outcome categories RP, PW and NP typically result from decisions across two dimensions, Responsive v. Not Responsive and Privileged v. Not Privileged.
- the RP outcome is assigned to every item coded +Responsive and ⁇ Privileged
- the PW outcome is assigned to every item coded +Responsive +Privileged and NP assigned for items that are ⁇ Responsive.
- some embodiments add an arguable category; most often it is added only for the Responsive-Not Responsive dimension, as the Privileged decisions generally use more black and white criteria.
- an “arguable” state is useful for at least two reasons. Coding decisions made earlier in the process may involve more uncertainty, and the use of the “arguable” code provides a means of quantifying the range of possible variation introduced by that uncertainty. As the review progresses, these items are, in one embodiment, revisited to see if any coding decisions should be overturned. In one embodiment, as the review progresses all of the reviewed items are revisited. Secondly, there will always be some items that are just difficult to categorize under any scheme. Providing the two options for how those items should be handled allows the litigation team to make a choice between the alternatives. Typically, if arguable items are considered responsive then the coverage scores improve and the over-generation scores get worse (i.e. higher).
- coverage is the percentage of actually responsive items that were produced, while over-generation is the ratio of total items erroneously assigned to a production category over the number of items reviewed to be in that category.
- some embodiments may utilize more than one “arguable” state, and may assign a variety of user interface labels to them, for example: “responsive but opinions may differ” and “non-responsive but opinions may differ.”
- the analyst will construct the initial set of categorization mechanisms during the determination of the supra-responsive query and the first round of sampling review.
- existing categorization mechanisms from related prior work on the same corpus may be reused.
- the actual categorization mechanisms used may come from anywhere, be of any form, and may also be reused, both at a class and instance level.
- categorization components are constructed over the entire reviewable universe.
- the system will generate a set of rules that assign each item to a production category based on the categorization components which contain it.
- the system implements algorithms that find a set of rules assigning production categories to items that are consistent with reviewer coding decisions. Since its not possible in general to find a categorization that is perfectly consistent with reviewer coding decisions, the system searches for rule sets that produce high levels of agreement while also protecting against a problem called “overfitting.”
- consistency is measured by comparing the categories assigned by the rule set against the categories assigned by reviewers, using the over-generation and coverage measures.
- the system will find groups of categorization components that, when intersected, result in a set of items which all share the same reviewer coding decision. If groups could be found that collectively contain all the items in the reviewable universe, one could then make a set of rules for categorizing the reviewable universe by extending the reviewer coding decisions given to the reviewed items in each of these intersection sets to the rest of the unreviewed items in each intersection set. Finding a group of intersection sets that are both consistent with reviewer decisions and collectively contain all the reviewed items is sufficiently unlikely that it can be considered a practical impossibility.
- intersection sets It is also extremely unlikely to find intersection sets which don't overlap with each other. Thus there will be conflicts and a relative priority between intersection sets is needed.
- the method implemented in this invention finds a group of intersection sets that are highly consistent with reviewer decisions, resolves conflicts and balances against the problem of overfitting (i.e. categorizing based on features in an individual item instead of a class of items).
- the system builds a list of rules that assign a production category to items that occur in the intersection of categorization components.
- a rule might specify that any item contained in all 3 categorization components should be labeled as RP.
- Rules in the list are evaluated in sequential order, top to bottom. The first rule that succeeds determines which category is assigned.
- This ordered list of rules is called a production rule assignment, or simply a production assignment. The method defined below seeks to choose a production assignment that assigns production categories which are consistent with reviewer decisions while avoiding overfitting to the reviewed data.
- Overfitting occurs when the production category assigned to a set of items agrees with the specific reviewer decisions, but the combination of components—or the combination of components when at that position in the rule list—does not accurately represent a significant number of items in the larger universe. Especially for rules that occur farther down in the list, many of the items they would normally have captured have already been filtered out and the chances of an arbitrary correlation to features of a specific set of items become more likely.
- the system's goal is to find the smallest number of rules which capture relatively large sets of items that are consistent with reviewer decisions. This is because the commonalities in a larger set of items are more likely to be meaningful. Suppose, for example, that a rule was created for a set of one item.
- That item may have been erroneously placed in one or more of its categorization components, as any categorization techniques will have exceptions. In general, that item may be better or worse representative of the categories into which it was placed. Therefore the confidence in the categorization decision is low for any of the unreviewed items that the rule captures. When a rule has been found valid for a large set of reviewed items, on a statistical basis the confidence in its decisions is much higher. In one embodiment, the system implements a measure for determining the likelihood that a production assignment has become over fitted.
- the search space of all possible production assignments is rather large. Given a set P of production category labels and a set C of categorization components, the set of all possible rule conditions is simply the power set of C, which has size 2
- the consistency of a production assignment with reviewer decisions is measured via coverage and over-generation Figures, as these can be meaningfully projected to the turnover set.
- the system uses a lattice structure [ 9615 ] in order to mathematically represent a partial ordering (via set inclusion [ 9618 ]) of groups of similar items, called itemsets [ 9629 ] from which categorization rules [ 9619 ] may be generated.
- itemsets [ 9629 ] from which categorization rules [ 9619 ] may be generated.
- other embodiments might instead select another type of mathematical structure which provides similar properties.
- the invention does not depend on the specific representation so much as the ability to efficiently exploit the partial order.
- a lattice representation is a straightforward representation of a partial order.
- Itemsets as represented in the lattice have a parent [ 9607 ], which is the “immediate successor” [ 9613 ] of the itemset.
- an immediate successor is any itemset y>x for which there are no other itemsets z such that y>z>x.
- itemsets have an “immediate predecessors” [ 9614 ] which are considered children [ 9608 ] of the itemset.
- Itemsets have a consistency [ 9623 ] score which measures the degree to which reviewer coding decisions on their members are similar.
- the system computes closed itemsets [ 9622 ] in a matrix consisting of items, the categorization components containing those items, and the reviewer coding decisions for each item.
- FIG. 98 depicts such a matrix in block [ 9805 ].
- the computation of closed item sets is used in approaches such as Data Mining and Formal Concept Analysis. There are several widely known algorithms for doing this computation, which may be utilized by different embodiments.
- a formal context K consists of:
- a pair (i, a) in R expresses that the object i has the attribute a.
- a formal context is equivalent to the matrix [ 9805 ] mentioned above.
- the attributes will consist of the set of categorization components that are being fitted. This describes a matrix, which is augmented with an additional set of attributes representing the production category labels that are derived from reviewer coding decisions. As described previously, there may be more than one combination of production category labels for which the system computes results, i.e. the ⁇ ARP, APW, NP ⁇ and ⁇ RP, PW, ANP ⁇ sets described above. In some embodiments, these additional attributes are not used in the closed item set computation, but as described later are rather used to derive additional properties stored for item sets found during the computation.
- an item set s is a pair (X, Y) [ 9850 ], for X a subset of I and Y a subset of A, such that for all items i in X and attributes a in Y, (i, a) are in R.
- X is called the extension [ 9855 ] of s
- Y is called the intension [ 9860 ] of s.
- this can be visualized by rearranging the rows in the matrix so that all rows in X are contiguous and all columns in Y are contiguous [ 9810 ], it can then be seen that all matrix elements in the region of their intersection are set to 1 [ 9815 ].
- f(X) defines an intersection of the set of all attributes that are shared in common between all items of X.
- the blocks [ 9815 , 9820 , 9825 , 9835 ] are examples of some of the closed itemsets in [ 9810 ].
- the system determines if TOP is itself a closed item set with a non-empty attribute set Y. If so, then the attributes in set Y of TOP are all underspecified, meaning that the presence or absence of those attributes cannot be used to subdivide the list of items into smaller categories. Thus these attributes can essentially be ignored for purposes of further analysis. This is equivalent to saying that a categorization component that captures everything is not very useful.
- n is the number of items that will be used in construction of the rule set.
- the size of the itemset is difficult to characterize as it completely depends on the data, however for sample sizes of a few thousand items and a few dozens of categorization components, a typical expectation might be that there are at most a few hundreds of item sets used if not much less.
- an implementation of the closed item set algorithm is augmented to associate additional values with each computed item set.
- Tallies of item counts in each production category mapped from the reviewer codes are computed during the phase in which the membership of an item set is determined. This has very little extra cost, as entries in the matrix of item attributes must anyway be scanned at some point in the normal course of the computation.
- An additional pair of flags is associated with each item set indicating whether or not the children of an item form a set cover over it and whether the item set is similarly a member of a group of children that form a set cover on their parent. If the union of a group of sets C.sub. 1 , C.sub. 2 , . . .
- C.sub.n contains a set P then the group of sets C.sub. 1 , C.sub. 2 , . . . , C.sub.n is said to cover the set P.
- the system computes the intersection of the item lists of its children. As there must be a phase in the closed item set algorithm that finds the intersection of the item lists for combinations of components, this additional calculation can be added at that point with relatively minimal additional cost.
- the calculation of the covering flag and the intersection set could be computed in an extra pass over the lattice (and in this case only a subset of the item sets need to be examined).
- the system automatically selects a “fence” of item sets from the lattice, depicted in FIG. 98 , block [ 9870 ].
- the fence is a sequence of item sets that cover the space of sampled items. The intent is to find the largest item sets for which reviewer coding decisions are highly consistent. In some embodiments, the consistency of an item set is measured by tallying up the reviewer coding decisions and then taking the ratio of the largest tally to the total size of the item set. However other embodiments may opt for different measures. Some embodiments will calculate multiple ratios, or the best fit over all alternatives introduced by adding arguable categories.
- the system selects items to be placed in the fence. In one embodiment, this is done during a (partial) depth-first traversal of the lattice. Starting with each child of TOP, the system takes any item set that is sufficiently consistent, (i.e. has a ratio of above a minimum threshold), places it in the fence, and stops traversal on that branch. Otherwise, if its children do not form a set cover over it, the system adds the item set to the fence and terminates the search on that branch. Otherwise, the system continues the search for each of the children. Note that when item sets have more than one parent in the lattice, it can create situations where both a parent and one of its children are in the fence; the parent being reachable on one path and the child reachable on a separate path.
- the system now computes an initial order on the items in the set.
- the system simply sorts itemsets by their consistency scores. This may or may not produce the most optimal ordering for a particular choice of fence. However the extent to which it can be suboptimal is limited. This is because in general the largest changes to the categorization will come from the inconsistent item sets, which are already sorted to the end of the list. We can never gain anything by placing an inconsistent itemset before a more consistent itemset because as a general rule, placing a less consistent itemset before a more consistent one will result in the assignment of a larger number of inconsistent items. However there may be isolated cases in which placing the more inconsistent item earlier does result in a better overall categorization. This might happen if earlier rules in the list catch the sorts of items that cause inconsistency in that item set. For this reason the system provides further tools for analyzing and adjusting the ordering of itemsets in the fence, as described below.
- the method described above provides a fast and efficient method for obtaining a good rule order, however the invention does not depend on a particular choice of ordering.
- the crucial issue for this part of the invention is the conversion of a search problem in a very large search space (i.e. finding a production rule assignment with good performance) into a representation that can be used to directly generate a specific and targeted set of rules.
- closed item set representation is used.
- the method applies equally well to representations that are equivalent to closed item sets, and for that matter generalizations of closed item sets, such as clusters formed by relaxing the constraint that all attributes must be shared in common between items in an itemset (different data clustering techniques and block modeling from social networking theory fall in this category for example).
- any method that identifies subsets based on high similarity of their attributes is appropriate; all that is required is the ability to place a meaningful partial ordering on the result set.
- a partial order is meaningful in this context so long as there is an inverse relationship between the number of items in an itemset and the intension of the set.
- the intension is directly measured as the set of attributes, and additional attributes in the intension implies more specificity.
- Other embodiments may have alternate representations of intension, so long as the representation used to measure similarity of items can be related to specificity of the description.
- the method by which we take advantage of any partial ordering that has the characteristics described above utilizes a fence.
- the procedure for finding a fence involves an implicit balancing act between finding large itemsets that have been reviewed relatively consistently and are not prone to overfitting, against finding smaller itemsets that have higher consistency scores but are more prone to overfitting.
- the itemset/lattice representation allows us to find the largest consistent itemsets quickly as they appear high up in the lattice and the level-wise expansion of the fence enforces the other side of the tradeoff. As we expand the membership of the fence, the size of itemsets necessarily shrinks.
- the production rule assignment directly corresponds to the ordered fence of itemsets.
- the rule antecedents are based on the itemset intension.
- the rule antecedents are simply a list of set membership tests for each of the categorization components represented by the attributes in the intension. Any item that is a member of all listed categorization components is matched by the rule.
- the rule consequent is the production category that will be assigned to the item. As noted above, the production category is mapped from the reviewer coding decisions that best represent the itemset. In order to produce the final production categorization, the system scans the list of rules for each item in the reviewable universe.
- the system assigns the production category stored as the consequent of the rule and moves on to the next item. If an item is not matched by any rule, it is placed in a remainder pool. In some embodiments, the pool may be separately reviewed in its entirety in order to determine the production categories for items contained in it. In other embodiments, the remainder is first sampled by the analyst to determine whether or not a new categorization component could be added to capture some part(s) of it. If new categorization components are added, the system performs additional rounds of analysis.
- the system iterates through the list of sample items and evaluates the rule list as described above. In practice there are typically several variations of the score reported. If the review categories include “arguable” codes, then in most embodiments multiple scores will be calculated. In the example scenario described above, the overall measures would be calculated for the (ARP, APW, NP) and ⁇ RP, PW, ANP ⁇ sets.
- An additional option is presented, in one embodiment, which recalculates the consistency scores by counting only the items actually captured by a rule rather than the consistency scores for the components that the rule is based on. The tallies are not recounted for consistent item sets, because any subset of items captured by them is by definition consistent.
- the production category label is set according to the highest frequency coding decision out of those captured by the rule. Doing this is very likely to cause overfitting, but there may be cases where the smaller set actually captured by a rule, successfully identifies a useful class of items.
- the option is primarily provided as an exploratory tool in some embodiments.
- the recalculation of consistency scores can be restricted to only those nodes currently selected in the lattice view, assuming they are currently included in the fence.
- the system can in one embodiment generate permutations of the fence in order to score each of them and will update the fence to the best scoring alternative.
- the system does not need to explore the full combinatorial space however.
- the search space can be reduced, and in some cases quite drastically.
- purely consistent components components with a consistency at or above a threshold—which may in some embodiments be user-determined—do not need to be permuted with other nodes in the fence.
- the remaining item sets can be partitioned into non-overlapping subsets. Since there are no items that can be captured by rules in more than one of these partitions the system only evaluates permutations of item sets within each partition. There are various methods that can be used to partition the item sets, and different embodiments may utilize different ones.
- a simple technique for finding the partitions is based on using the matrix as an inverted index that associates reviewed items to all the item sets that contain them.
- the algorithm will build a set of cluster nodes, each representing a potential partition in the fence. If two partitions are found to overlap, the clusters will be merged.
- a cluster node contains a reference to a parent cluster node, a list of children and a reference to an itemset.
- the algorithm uses a lookup table to keep track of whether nodes have been encountered previously and a set of root cluster nodes.
- the system scans through the remaining itemsets in the fence. For each item set, the system creates a cluster node with a reference to that itemset, adds it to the root set and then scans through the item set members.
- the system adds an entry containing the cluster node for that item set. If an item set member is not currently in the table then the system adds an entry containing the cluster node for that item set. If an item set member does have an entry in the table then the system walks upwards through parent references until it finds a root cluster node, if the found root is not equal to the current node, it then adds the root cluster node to the current cluster node's child list, sets the parent reference for the found root node to the current node and removes the old root node from the root set. The parent of the cluster node retrieved from the table can also be updated at this time as a small optimization. After all item sets have been scanned, the remaining members of the root set each represent one partition, from which the member item sets can be recovered by walking through the child lists.
- a visual analysis tool displays the item sets selected to be in the fence by drawing a literal fence over the respective nodes in the lattice view.
- FIG. 99 An exemplary implementation of this is illustrated in FIG. 99 .
- it contains a visual representation of a lattice [ 9970 ], with the fence clearly indicated by a line [ 9950 ] whose style differs from the line style used to render the lattice.
- item sets are decorated with pie icons [ 9960 ] that represent the consistency score calculated for the items in this item set.
- clicking on the icon for an item set brings up a floating menu [ 9955 ] with different options.
- Some embodiments may also include a tree or hierarchical parent-child view of the item sets in a detail list view [ 9975 ].
- the user may drag the icons representing fence posts [ 9945 ] on item sets upwards or downwards in order to change the membership of the current fence.
- the partitioning of this fence discussed above namely the set of consistent item sets and the equivalence classes of overlapping subsets from the remaining elements in the fence, are shown in some embodiments by bounding boxes and color highlighting.
- Some embodiments may provide additional controls that allow interactive optimizations, or that control different aspects of the visualization of the lattice.
- some embodiments allow a global setting whereby itemsets below a certain size are excluded from all calculations [ 9905 ], as a simple method for avoiding overfitting.
- Categorization components represented by attributes contained in the intersection of all child item sets of a parent are underspecified with respect to the parent item set, as they cannot be combined with any other components to make any kind of categorization decision in the parent item set.
- Categorization components that are underspecified, particularly with respect to large item sets are candidates to be broken out into subcomponents in subsequent iterations.
- the user can request a check to find underspecified attributes with respect to the selected itemsets [ 9990 ].
- the intent is to split the list of items hit by that component so that the new sub-components cross with existing item sets to create new, potentially higher consistency item sets.
- a check is also performed to see if the intersection of all item sets in each partition of the fence is non-empty [ 9985 ]. Any components so identified are the most likely candidates for removal in subsequent iterations. However as shown below there is an additional test that can help to determine whether it might be useful to attempt splitting the component as for the parent-child case.
- Each child item set within the lattice adds at least one attribute over its parent; this follows from the definition of a closed item set.
- the consistency of children item sets will tend to be higher than that of its parent, though this is not always the case.
- the cases where children do become more inconsistent, or where the consistency of some children remains lower than those of other children of the same parent highlight areas for investigation when performing the next refinement round on categorization components.
- itemsets that have a lower consistency than their parent are highlighted [ 9995 ], as well as itemsets that have a consistency similar to their parent but much lower than other children of that same parent.
- “much lower” is defined to be one standard deviation less than the average consistency for all children of the parent. Embodiments may use various measures and heuristics for identifying these cases.
- the set of attributes associated with one of these marked inconsistent sets are also of interest, particularly the additional attributes that appear in one of these child itemsets.
- these attributes are listed in hover text that appears when the mouse icon is held over one of these highlighted item sets.
- This last category is interesting in that a new component could be added in order to “patch” the scheme. This has the intent of producing higher consistency item sets derived from the new component in the next iteration. Therefore in one embodiment, itemsets generated from the “patched” component will be placed before the inconsistent item set in the fence constructed for the next iteration.
- One metric that is particularly useful for analyzing an inconsistent item set is based on the set of attributes it adds over its parent or multiple such sets of attributes if it has multiple parents.
- the system calculates the consistency score for the union of all item sets that differ from their parent by that same attribute set.
- a detail list view can be generated that lists all these item sets [ 9980 ], as well as their respective consistency scores and an overall score.
- the tool uses heuristics based on these observations to generate warnings and advice through the user interface. As the number of items involved in an item set decrease, the chances are greater that the characteristics common to those items are unique to those items rather than representative of characteristics common to a general class of items. The iterative nature of the categorization process will help to guard against this class of overfitting.
- the procedure for selecting the fence is designed to pick a reasonably well fitted production assignment rule scheme while also protecting against overfitting to reviewed samples.
- the fence allows analysts to use a larger number of specifically focused categorization components. However there may be fences that produce a better fit. Therefore an analysis tool allows the analyst to modify the fence interactively. An analyst may want to modify the initial fence in order to find a better fit, or as an exploration tool with which to understand how components interact.
- the tool allows an analyst to explore modifications to the fence by selecting a point in the fence, (a ‘fence post’), and dragging it either upwards to the position of a parent or downwards to that of a child. Either action results in a modification to the fence, either the addition of a parent item set and the removal of its children item sets or the addition of children item sets and the removal of their parent item set as appropriate.
- a fence post downwards
- the effects are determined by iteratively analyzing each successive level the post is moved through. For each child level, fence posts are added to all of the children of the parent at the previous level, and the parent fence post is removed.
- one path is selected by the system.
- the path followed through the expansions on each successive level is selected.
- the user selects the path in this event.
- the semantics of dragging a fencepost upwards also takes into account whether or not a node has multiple parents in the lattice.
- an additional fence post may be left on the originally selected node after dragging upwards. This is done to ensure that no breaks are introduced into the fence.
- the parent node is first added to the fence and then all paths leading upwards from the originally selected node (via parents) are checked to see if they all lead to a node in the fence. If at least one path does not contain an ancestor node in the fence, a fence post is left on the originally selected node. If the selected fence post is dragged upwards by more than one level, this process is applied iteratively on all paths leading from the original node upwards.
- the derived properties described in prior sections are also incrementally recalculated to conform to the new fence.
- the regeneration of the assignment rule list is an exception.
- the rule list is updated, but the ordering of rules is only partially updated.
- new rules corresponding to the new child item sets added to the fence are placed in a sequence and sorted based on their consistency scores. In one embodiment, this sequence is then spliced into the rule list at the position of the rule they are replacing.
- the user can request that permutations of the rule list are to be recomputed as a separate action, as this computation may take some time. When collapsing points upwards, the rules corresponding to the current children are removed.
- a new rule is inserted at the lowest position in the set of rules removed.
- a summary currently being reported will be updated if affected.
- the user can also request an automated version of these procedures [ 9935 ], in which all alternate versions of the fence that can be formed by selecting one point in the original fence and moving it upwards or downwards by one level are computed.
- the fence is updated to the best scoring alternative, derived properties are recalculated as above and the detail list view is updated.
- the automated search is constrained to only the single point variations in order to limit computation time.
- the tools are provided to help the analyst to explore the space in order to determine where best to make additional refinements to the categorization components. Unfortunately even within the vastly restricted search space provided by the item set lattice, an exhaustive search is still prohibitively computationally expensive.
- the system does not prohibit the analyst from dragging a fence post down from a parent which is not covered by its children. Since there are now items that may not appear in any of the itemsets in the fence, coverage numbers may fall. However, moving the fence downwards may be a way to identify more consistent subsets of a highly inconsistent parent and the new production rule assignment that results may also have a lower over-generation score. The improved over-generation score may be worth the trade off in reduced coverage.
- the system provides a measure for determining whether the set of categorization rules is overfitted to the reviewed data.
- a projection of the final categorization based on the results of applying the categorization rules to the sample is compared to the results of applying the categorization rules to the dataset as a whole.
- the projection is based on fairly straightforward arithmetic, and takes into account the stratification of the universe from which the samples were drawn. For each of the original strata the system collects tallies for each of the possible reviewer calls consisting of the number of cases where the categorization agrees with the reviewer call, versus the number of disagreements. The reviewer calls map to a result category in a natural way.
- the reviewer calls map one to one onto the result categories.
- S.sub.n with counts S.sub.n.ac.sub.m and S.sub.n.dc.sub.m for the mth reviewer call and a total count c.sub.m for the mth call
- the projected categorization is
- the total projected turnover is the sum over all strata and all reviewer calls.
- this statistic is not re-computed interactively as it typically involves scanning through a large database of items, however it can be requested as a separate action in most embodiments.
- Each matter is assumed to have a set of objective qualifications for responsiveness—for example, that the items in question are associated with certain individuals, and fall within a certain date range, or contain certain words.
- objective qualifications for responsiveness—for example, that the items in question are associated with certain individuals, and fall within a certain date range, or contain certain words.
- a report may be generated noting differences in reviewer decisions when similar documents do not differ with respect to objective qualifications, since these are the review decisions which may deserve a bit more scrutiny.
- attachments A more difficult version of this problem involves attachments, as the responsiveness state of an attachment may be governed by its parent email rather than its own content. Since attachments may or may not be coded separately from their parent emails, a separate decision in each case on the basis of the individual attachment may not always be directly inferable.
- the system provides the reviewer-user with information about how this item and any proper subset of it that is contained elsewhere has been reviewed. In this way, the reviewer or the administrative user can decide when a document no longer needs to be re-read for the Nth time.
- this is done in the following manner, as shown in FIG. 72 :
- the system displays a tally of the results.
- this display is available form the main review menu, and the existence of such a tally is visually indicated to the reviewer.
- the names of the reviewers who made the review decisions are also displayed, or are directly available from this tally.
- the system likewise shows a tally of the review decisions.
- a unanimous decision of prior reviewers is needed to discard.
- the behavior may be configured by the user.
- all items that contain items (including themselves) that have passed a certain threshold of unanimity in one or more of these metrics can be specified for removal from the review pool. For example, an administrator could decide that if a particular item had a) never been reviewed as responsive in any kind of standalone form, and b) had similarly not been reviewed as responsive in the context-dependent metric, and had been reviewed a total of 10 times within these two measures, this item could be clearly labeled in the UI display as being “NR” so that reviewers would not waste more time re-reading it.
- the prior decisions could be considered as a “hint.”
- a hint One possible embodiment of displaying such a hint [ 334 ] is illustrated in FIG. 3F .
- the display of the tally allows reviewers to use their own judgment whether to invest the time to re-read the item.
- whether or not the “hint” icon [ 334 ] is displayed is user-configurable.
- the hint icon [ 334 ] may be hidden when the document involved is particularly sensitive in terms of topical content, or actor (such as the CEO), and there is therefore a desire to ensure a full and thorough contextual review.
- the administrative user may determine that only some groups of reviewers, or individual reviewers may be allowed to see such hint icons [ 334 ].
- quoted text from message A in message B may be displayed inline or as an attachment, depending on the settings of the email reader.
- the “proper subset” logic noted above applies to emails as well. This is to say that if message A was found to be responsive, message B must also be found to be responsive, unless precluded from being so by objective factors, such as falling outside the prescribed date range, or being associated with a privileged actor.
- a warning indicator is present that, when clicked on, displays the nature and date of the change.
- Some embodiments will also note directly conflicting human reviewer decisions on the same item at different stages in the workflow. In some embodiments, this is done via one or more of: an email alert, a reports of such items, a special icon, and in still others, any such “reversed” or disputed items are automatically placed in a special area in workflow for further examination by a more senior user. In some of these embodiments, different behaviors may be established by the user based on the properties of the item.
- FIG. 73 One embodiment of such folders is illustrated in FIG. 73 .
- Items may be placed in these folders either individually, by some sort of bulk copy (such as copying the members of one folder into another), by performing a query, or otherwise selecting items, and depositing the results in a particular folder.
- a “inding” [ 7415 ] is a folder whose contents have been (or will be) determined by a user to be pertinent or useful to the specific topic or topics that are associated with the finding [ 7415 ].
- the semantics of the findings folder [ 7405 ] are such that it is expected to contain at least one critical item or result that supports a definitive conclusion. In one embodiment, this determination may be made by a user, the system, or both.
- a findings folder [ 7405 ] requires an explicit hypothesis [ 7425 ] to be associated with it whose truth or falsity can be established.
- findings folders may have a number of different properties associated with them, including but not limited to, the following: name, associated hypothesis, query, creation date, owner, one or more queries, current “truth” state, set of possible truth states, other folders it contains or has been merged with, and an audit log of all additions, deletions, and modifications to both the structure and contents of the folder.
- Some embodiments may have additional system attributes such as the user permissions that are associated with the folder [ 7405 .]
- a findings folder [ 7405 ] differs from all other folders since it is designed to contain only items that relate to proving or disproving a particular hypothesis. This differs from other folders that may be serving as temporary holding areas for items which are to be reviewed later, or that exist for some specific reason other than proving or disproving a hypothesis. However, once the items that are assuredly relevant (because they are in one or more finding folders [ 7405 ]), have been identified, Venn diagrams and other comparisons that relate the contents of the different findings folders [ 7405 ] together can be automatically generated. This may yield important and non-obvious insights about the relationship of different aspects of the investigation.
- “finding” items can be clustered in different dimensions (using a multidimensional scaling model or otherwise) to determine previously unsuspected similarities between the items in different findings folders [ 7405 ].
- FIG. 75 One embodiment of this is illustrated in FIG. 75 .
- algorithmic and/or statistical post-processing can be used to further extend the work performed by the human investigators.
- the system in one embodiment may reorganize the findings folders into a hierarchical representation that reflects literal data containment relationships (as caused, for example, by a user dragging a large number of items into a findings folder that makes it highly duplicative with another findings folder.)
- the user specifies the degree of overlap that triggers this. This is illustrated in FIG. 76 .
- strict containment may not be enforced.
- the destination folder is augmented with an icon [ 7605 ] to indicate that another folder has been merged into it; in some of these embodiments, this icon [ 7605 ] is only displayed for a user-configured period of time, as at some point, the merge becomes old news.
- the system may automatically query the user, asking whether two or more findings folders should be merged into one based on the degree of overlap of contents.
- a user can indicate that she wishes to merge two folders by dragging one folder on top of the other.
- administrative users can sign up for alerts related to specific events, including but not limited to: user attempts to merge folders that do not obey the configured parameters for merges, administrator-configured overlap of items among different users, items being removed by one user from a findings folder set up by another user (if this is allowed).
- tabular reports with user configurable header columns [ 7805 ] are generated, which visually indicate which different users [ 7810 ] filed the same items in their findings folders. Note, however, that other embodiments or configurations may display additional or different data.
- FIG. 79 illustrates one embodiment of this that uses a bar chart visualization to indicate the overlap between users.
- mousing over one of the bars [ 7910 ] will bring up a count of the items in question in a display [ 7915 ].
- Some embodiments may only offer pairwise comparisons of users, while others will group as many different users as appropriate together in a single bar, based on degree of overlap.
- higher level reports [ 8010 ] on the contents of findings folders are available which provide information on groups of items based on a queryable set of attributes rather than just on individual items.
- groupings may include, but are not limited to: items in the same discussion, in the same cluster, or of the same specific type, such as a particular type of periodically issued report.
- the method described in the SOCIOLOGICAL DATA MINING APPLICATION is used to identify and name of each group; in some embodiments such groups may also be queried. As shown in FIG.
- each row [ 8015 ] of information indicates a set of documents, rather than an individual document; all information after the first column contains aggregate information for all documents of the described type. Providing this information at a higher level allows the user to see the bigger picture, as opposed to getting mired in a sea of individual documents. This style of report may be used to assess findings folders contents, or as part of any more general query.
- some embodiments may also contain a visual timeline [ 8020 ] to help illustrate to the user the period of time that is collectively covered by each set of documents [ 8015 ].
- the timelines [ 8020 ] also includes all discussions involving the items in question.
- the timelines [ 8020 ] may only display the items themselves.
- timelines [ 8020 ] appear in the table instead of the spelled out dates.
- Different embodiments may employ different embodiments or styles of the timeline [ 8020 ] view.
- Some embodiments may provide grouping and sorting controls to control the display of the timelines [ 8020 ].
- the actor names may be replaced with photographs or other images to represent the individual actor.
- a filter in order to handle very large sets of documents, a filter may be provided.
- a filter [ 8005 ] is a control used to further refine the query, or limit the documents that will be displayed.
- a folder into a “dock” [ 8110 ] (that is, a specified area for receipt of dragged objects) within a query window [ 8120 ]
- the user may indicate that she wishes to limit the query to items in this finding folder (and others that have likewise been specified.)
- a “dock” [ 8110 ] that is, a specified area for receipt of dragged objects
- the user may indicate that she wishes to limit the query to items in this finding folder (and others that have likewise been specified.)
- FIG. 81 One embodiment of this is shown in FIG. 81 .
- New relationships between actors may be inferred on the basis of the presence of items related to them that appear in different findings folders.
- visualizations of actor communication could be limited to either the items that are currently in finding folders, or extended to include other items similar to those currently in one or more findings folders. Similarity may be judged by a number of different metrics, such as topic clustering, actor clustering (clustering on the basis of the actors involved) or membership in the same discussion.
- finding folders may also be assigned ordinal priorities [ 8205 ] by the user. This information can be used both to order the display of folders on screen, and to de-duplicate items that may appear in multiple folders. This is logically illustrated in FIG. 82 . For example, if a particular item of interest appeared in 18 findings folders, if the user were to select the “unique” command [ 8210 ], the item will only show up once, in the findings folder that had been assigned the highest priority [ 8205 ] by the user. In one embodiment, the item is completely removed from any other folders. This is useful because without such a feature, the same item may be looked at again and again.
- the user may designate specific items in a discussion or particular text in a document as being of especially high interest by selecting this text with the highlighter button [ 8305 ], thus creating an instance of highlighted text [ 8320 ], as illustrated in FIG. 83 . In many embodiments, this is done through a highlighter metaphor.
- a highlighter “notes” panel [ 8315 ] may automatically pop up when the user selects text with the highlighter cursor [ 8325 ]. In one embodiment, the highlighter “notes”[ 8310 ] will appear upon mouseover of the highlighted text.
- This information is used to both help prevent false unifications as well as to determine similarities that might otherwise stay buried. For example, if two entirely different passages are considered relevant to completely different topics in a 200 page document, the presence of the document in the two different folders should not necessarily be considered overlap.
- the highlighting also provides the user with a means of realizing that, for example, the same paragraph in the 200 page document was material to resolving two apparently unrelated issues. Without such a mechanism, there is no way to identify which portion(s) of the item are the relevant ones.
- whether or not to consider each instance of highlighted information as a separate object for user review is a configurable setting. If such highlighted information is to be considered distinct, in some embodiments, any highlighted text in a document will be separately indexed as a new object in the index so that it may be correctly compared to other potentially relevant items in clustering and other analysis. If the highlighted text is a simply an item in a discussion, only the item and not its parent discussion will be similarly treated. This is illustrated in FIG. 84 .
- Findings folders may be profiled along different dimensions with a single click, using the profile button [ 8505 ], as shown in FIG. 85 .
- Different embodiments may have additional or different dimensions. Some of the dimensions which may be available include, but are not limited to: date ranges of the items, topics present, foreign languages present, named entities referred to, types of individual activities documented. Note that different combinations of dimensions may also be used.
- Different embodiments may employ different visualizations to convey similarity, and dissimilarity of data contents. Once the desired dimension(s) and visualization(s) have been specified by the user, profiling can be done with a single click.
- direct visual content comparisons can be made, in this instance a Venn diagram [ 9405 ] based on a degree of overlap.
- graphical querying is supported, as shown in FIG. 94 .
- clicking on any part of the Venn diagram [ 9405 ] selects or de-selects items that are represented by that portion of the diagram, for the purposes of a query.
- a finding posed as a yes/no question may be linked to a graphical visualization tool which allows users to both create and view decision trees, as shown in FIG. 87 .
- the user starts this process by creating one or more “question” nodes [ 8705 ], with the desired names, and then dropping them on the canvas [ 8710 ]. He also drops decision nodes [ 8715 ] onto the canvas [ 8710 ] and connects the different kinds of nodes together with the connector tool [ 8720 ] which results in the rendering of lines between the selected nodes.
- the basic mechanics of this are very similar to commonly used graphics packages.
- a simple example of a decision tree in this context is one in which if the answer to the above sample question is “yes”, the action is to try to plead out the case, but if the answer is “no”, then proceed to trial.
- decision trees as shown in FIG. 87 , may be, and often are, arbitrarily complex.
- a finding may have the property of being one of: true, false, or as yet unknown. All findings start off by definition in this last state. However, as information is added to the folder, the state may change for any of the following reasons, depending on the exact embodiment of the invention:
- One or more items are returned in response to a specific query, thus satisfying either the “yes” or “no”. For example, if the question associated with the finding folder is: “Did Joe Smith receive information about a proposed merger prior to May 5, 2001?” and evidence of at least one such email is discovered, the answer is “yes.”
- results of some kind of analysis other than queries have returned a definitive result—for example, a statistical analysis that definitively suggests that evidence has been tampered with.
- a statistical analysis that definitively suggests that evidence has been tampered with.
- FIG. 50 One example of statistical analysis, based on back-up data, is shown in FIG. 50 .
- the system may allow for additional states, such as “probably true” or “probably false.”. Note that in some embodiments, only users with the appropriate permissions may make such changes.
- the decision tree may be visualized using any graphing tool. Not all nodes in the tree must correspond to yes/no findings, However, when the user drops a new node on the canvas [ 8710 ], in some embodiments, a dialog [ 8725 ] will pop up asking the user whether the node should be considered a finding, and thus have a related findings folder [ 8730 ] created for it.
- double clicking on a question node [ 8705 ] in the decision tree [ 8705 ] view will bring up the contents of the associated findings folder.
- a query window is available that allows the user to specify all available properties of a query, including all of the characteristics of the returned view(s.)
- the administrator can configure query windows with reduced or specialized functionality for different types of users.
- FIG. 88 One embodiment of a more complex query window is depicted in FIG. 88 . As pictured, it allows the user to select all of the following:
- the actual query to be performed on the relevant objects [ 8810 ].
- Specifications include, but are not limited to, topic, tone, date range, and (not pictured) document type, specific keyword content, and various kinds of metadata.
- This option provides query controls in the generated window so as to facilitate query refinement, and one large view which contains the query results in whatever results format.
- Time Elapsed Style This will divide the interval of time selected in the query into equal time slices, providing both individual snapshots of each interval of time and a mechanism for “playing” the slides in sequential fashion.
- a default time slicing is provided, however the user may configure it globally and/or in the context of a specific query in most embodiments.
- Thumbnail Style This is somewhat similar, but displays small “thumbnail” images of different time slices in a grid formation.
- Tabular Brings up the query results in a table widget that allows different kinds of sorting
- View Styles [ 8825 ]. Most types of objects in the system—and in some cases even their properties—have a variety of possible ways in which they can be visualized. The options available for the user to select depend on the current selection state of other controls in the window. As pictured, sample styles range from basic charts to different visual styles of timeline, and different styles of communication graphs.
- the “Display in View” [ 8835 ] section indicates what types of information should be displayed in the query results, presuming that it is available. For example, a user could select a basic communication graph as the style of view [ 8825 ] that she wants, and then decide that she wishes to see personal vs. professional communication separately delineated from one another.
- this section may contain controls that allow the user to specify which of the available query controls should be brought up (and be associated with this particular query, if saved for reuse.) It may also include a control to specify the canvas tools are present.
- FIG. 90 illustrates the set of canvas tools that are available in one embodiment. To summarize, the options made available to the user in this section are capable of generating the different variations in controls, styles and layouts of the different visualization windows depicted in this application.
- the “Restrict to Folders” [ 8840 ] controls can be used in order to limit the query to folders, which may be selected through a folder browser or dragged and dropped into the “dock” provided.
- the “More options” button if clicked offers still further options, including but not limited to how the returned results should be sorted and grouped, and what line styles, fill characteristics, icons and other visual representations are to be used to represent the different types of objects that will be visualized in the returned result. Note therefore that in most embodiments, the legends for the visualizations are dynamically generated.
- the “save” button allows the query to be saved for future use.
- the information from these communication graphs can be presented in a tabular report form [ 8905 ], which lists each discussion that meets the requested criteria, and allows sorting on all columns.
- FIG. 89 illustrates one embodiment of a tabular view.
- Some embodiments may have a variety of columns, which may include, but is not limited to, one or more of the following: actors participating, discussion name, item count, discussion unique ID, day/night indicators for each item in the discussion, number of attachments, topics, foreign language content, and tonal content.
- Some embodiments may allow the columns to be user-configurable, including providing controls [ 8910 ] with which to hide those columns which are not currently of interest.
- the response to a query is some type of graphical depiction of all discussions that conform to the conditions of the query, each rendered in chronological sequence.
- Some embodiments may provide sorting controls, including but not limited to actor, topic, tone—or any combination of these—and ascending or descending sort over time.
- FIG. 91 Many of these graphical depictions are timeline views of one form or another, one style of which is illustrated in FIG. 91 .
- each type of communication medium is portrayed on its own separate line.
- the notion of discussion membership can be extended to other events or items of interest that occurred during the lifespan of the discussion, the travel of a particular actor [ 9110 ] in the discussion is illustrated. As illustrated, mousing over any of the boxes will, in this embodiment, bring up a floating box that provides more information about the relevant item or event.
- each item in the discussion is colored by media type [ 9220 ], for example, email vs. IM vs. phone call, and all types are displayed on the same line [ 9210 ] in order to conserve screen real estate. This representation is especially desirable when large numbers of discussion objects are retrieved, and will need to be displayed.
- media type [ 9220 ] for example, email vs. IM vs. phone call
- all types are displayed on the same line [ 9210 ] in order to conserve screen real estate.
- This representation is especially desirable when large numbers of discussion objects are retrieved, and will need to be displayed.
- FIG. 93 still other embodiments may label the blocks with appropriate icons [ 9305 ] or other indicators instead of just relying on coloration to initially distinguish the different types of items.
- Some embodiments may provide additional labeling or indicators [ 9310 ], including but not limited to displaying the primary actors for each discussion, and indicating key external events with vertical lines and labels [ 9305 ].
- each discussion object is labeled with the names of the primary actors and the name of the discussion, one embodiment of the determination of which is described in the SOCIOLOGICAL DATA MINING APPLICATION.
- SOCIOLOGICAL DATA MINING APPLICATION the SOCIOLOGICAL DATA MINING APPLICATION.
- Some embodiments also may group discussions together visually by user-specified properties, including but not limited to, topical content, the actors involved, the item or document types contained in the discussion.
- some embodiments may visualize different kinds of meeting related objects with different icons or other differentiating visualizations. For example, some embodiments may differentiate between requests for a call embedded in an email, versus a calendar event, versus an actual record of a phone call.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Library & Information Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
Description
Claims (20)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US12/422,902 US8135711B2 (en) | 2002-02-04 | 2009-04-13 | Method and apparatus for sociological data analysis |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US35440302P | 2002-02-04 | 2002-02-04 | |
US10/358,759 US7143091B2 (en) | 2002-02-04 | 2003-02-04 | Method and apparatus for sociological data mining |
US11/497,199 US7519589B2 (en) | 2003-02-04 | 2006-07-31 | Method and apparatus for sociological data analysis |
US12/422,902 US8135711B2 (en) | 2002-02-04 | 2009-04-13 | Method and apparatus for sociological data analysis |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/497,199 Continuation US7519589B2 (en) | 2002-02-04 | 2006-07-31 | Method and apparatus for sociological data analysis |
Publications (2)
Publication Number | Publication Date |
---|---|
US20090287685A1 US20090287685A1 (en) | 2009-11-19 |
US8135711B2 true US8135711B2 (en) | 2012-03-13 |
Family
ID=28045064
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US12/422,902 Expired - Fee Related US8135711B2 (en) | 2002-02-04 | 2009-04-13 | Method and apparatus for sociological data analysis |
Country Status (1)
Country | Link |
---|---|
US (1) | US8135711B2 (en) |
Cited By (17)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20090100039A1 (en) * | 2007-10-11 | 2009-04-16 | Oracle International Corp | Extensible mechanism for grouping search results |
US20110074713A1 (en) * | 2009-09-30 | 2011-03-31 | Sony Corporation | Remote operation device, remote operation system, remote operation method and program |
US20110119257A1 (en) * | 2009-11-13 | 2011-05-19 | Oracle International Corporation | Method and System for Enterprise Search Navigation |
US20140149363A1 (en) * | 2006-10-10 | 2014-05-29 | Adobe Systems Incorporated | Methods and apparatus for automated redaction of content in a document |
US8751520B1 (en) * | 2010-06-23 | 2014-06-10 | Google Inc. | Query suggestions with high utility |
US20140351275A1 (en) * | 2013-05-21 | 2014-11-27 | International Business Machines Corporation | Contributor identification tool |
US8997016B1 (en) * | 2011-01-04 | 2015-03-31 | Intellectual Ventures Fund 79 Llc | Methods, devices, and mediums associated with displaying an electronic pane |
US20170039246A1 (en) * | 2015-08-04 | 2017-02-09 | International Business Machines Corporation | Team analytics context graph generation |
USD786273S1 (en) | 2015-06-16 | 2017-05-09 | Adp, Llc | Display screen with graphical user interface |
USD800736S1 (en) | 2015-06-16 | 2017-10-24 | Adp Llc | Display screen with animated graphical user interface |
CN107291761A (en) * | 2016-04-05 | 2017-10-24 | 北京优朋普乐科技有限公司 | The matching process and device of a kind of regular expression |
US9984147B2 (en) | 2008-08-08 | 2018-05-29 | The Research Foundation For The State University Of New York | System and method for probabilistic relational clustering |
US10073926B2 (en) | 2015-08-04 | 2018-09-11 | International Business Machines Corporation | Team analytics context graph generation and augmentation |
US10546020B2 (en) | 2016-02-29 | 2020-01-28 | International Business Machines Corporation | Conversation purpose-based team analytics |
US10740406B2 (en) | 2013-12-06 | 2020-08-11 | Hewlett-Packard Development Company, L.P. | Matching of an input document to documents in a document collection |
US10783496B2 (en) | 2015-06-16 | 2020-09-22 | Adp, Llc | Balanced information system |
US11151171B2 (en) | 2018-04-27 | 2021-10-19 | International Business Machines Corporation | Accessing faceted information using ontologies |
Families Citing this family (94)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8943024B1 (en) | 2003-01-17 | 2015-01-27 | Daniel John Gardner | System and method for data de-duplication |
US8375008B1 (en) | 2003-01-17 | 2013-02-12 | Robert Gomes | Method and system for enterprise-wide retention of digital or electronic data |
KR100626267B1 (en) * | 2004-05-11 | 2006-09-20 | 엔에이치엔(주) | A system for visualizing a community activity, and a method thereof |
US8527468B1 (en) | 2005-02-08 | 2013-09-03 | Renew Data Corp. | System and method for management of retention periods for content in a computing system |
CN101194286A (en) * | 2005-03-24 | 2008-06-04 | 埃森哲全球服务有限公司 | Risk based data assessment |
US8032825B2 (en) * | 2005-06-16 | 2011-10-04 | International Business Machines Corporation | Dynamically creating multimodal markup documents |
US20060288309A1 (en) * | 2005-06-16 | 2006-12-21 | Cross Charles W Jr | Displaying available menu choices in a multimodal browser |
US20080189273A1 (en) * | 2006-06-07 | 2008-08-07 | Digital Mandate, Llc | System and method for utilizing advanced search and highlighting techniques for isolating subsets of relevant content data |
US7831928B1 (en) * | 2006-06-22 | 2010-11-09 | Digg, Inc. | Content visualization |
US8869037B2 (en) * | 2006-06-22 | 2014-10-21 | Linkedin Corporation | Event visualization |
US7797169B2 (en) * | 2006-10-13 | 2010-09-14 | Moore Barrett H | Method of facilitate obtaining, storing, and subsequently conveying information between related generations |
US8788620B2 (en) * | 2007-04-04 | 2014-07-22 | International Business Machines Corporation | Web service support for a multimodal client processing a multimodal application |
US8615490B1 (en) | 2008-01-31 | 2013-12-24 | Renew Data Corp. | Method and system for restoring information from backup storage media |
US8229081B2 (en) | 2008-04-24 | 2012-07-24 | International Business Machines Corporation | Dynamically publishing directory information for a plurality of interactive voice response systems |
US8214242B2 (en) * | 2008-04-24 | 2012-07-03 | International Business Machines Corporation | Signaling correspondence between a meeting agenda and a meeting discussion |
US10296557B2 (en) * | 2008-05-28 | 2019-05-21 | International Business Machines Corporation | Content creation system configured to provide feedback on content coverage |
US9852127B2 (en) * | 2008-05-28 | 2017-12-26 | International Business Machines Corporation | Processing publishing rules by routing documents based on document conceptual understanding |
US10169546B2 (en) * | 2008-05-28 | 2019-01-01 | International Business Machines Corporation | Generating document processing workflows configured to route documents based on document conceptual understanding |
US8380513B2 (en) * | 2009-05-19 | 2013-02-19 | International Business Machines Corporation | Improving speech capabilities of a multimodal application |
US8290780B2 (en) | 2009-06-24 | 2012-10-16 | International Business Machines Corporation | Dynamically extending the speech prompts of a multimodal application |
US8510117B2 (en) * | 2009-07-09 | 2013-08-13 | Nuance Communications, Inc. | Speech enabled media sharing in a multimodal application |
US8416714B2 (en) * | 2009-08-05 | 2013-04-09 | International Business Machines Corporation | Multimodal teleconferencing |
US20110078188A1 (en) * | 2009-09-28 | 2011-03-31 | Microsoft Corporation | Mining and Conveying Social Relationships |
US9111254B2 (en) * | 2009-11-02 | 2015-08-18 | At&T Intellectual Property I, L.P. | System and method to manage electronic data related to a legal matter |
US9053454B2 (en) * | 2009-11-30 | 2015-06-09 | Bank Of America Corporation | Automated straight-through processing in an electronic discovery system |
US8738668B2 (en) * | 2009-12-16 | 2014-05-27 | Renew Data Corp. | System and method for creating a de-duplicated data set |
US8296130B2 (en) * | 2010-01-29 | 2012-10-23 | Ipar, Llc | Systems and methods for word offensiveness detection and processing using weighted dictionaries and normalization |
US9785987B2 (en) | 2010-04-22 | 2017-10-10 | Microsoft Technology Licensing, Llc | User interface for information presentation system |
US20110282861A1 (en) * | 2010-05-11 | 2011-11-17 | Microsoft Corporation | Extracting higher-order knowledge from structured data |
US8786597B2 (en) * | 2010-06-30 | 2014-07-22 | International Business Machines Corporation | Management of a history of a meeting |
US9043296B2 (en) | 2010-07-30 | 2015-05-26 | Microsoft Technology Licensing, Llc | System of providing suggestions based on accessible and contextual information |
US20120084180A1 (en) * | 2010-10-01 | 2012-04-05 | Dowdell Catherine D | Personal Growth System, Methods, and Products |
US8687941B2 (en) | 2010-10-29 | 2014-04-01 | International Business Machines Corporation | Automatic static video summarization |
US8977979B2 (en) * | 2010-12-06 | 2015-03-10 | International Business Machines Corporation | Social network relationship mapping |
US8396871B2 (en) | 2011-01-26 | 2013-03-12 | DiscoverReady LLC | Document classification and characterization |
WO2012116287A1 (en) * | 2011-02-24 | 2012-08-30 | Lexisnexis, A Division Of Reed Elsevier Inc. | Methods for electronic document searching and graphically representing electronic document searches |
US9053750B2 (en) | 2011-06-17 | 2015-06-09 | At&T Intellectual Property I, L.P. | Speaker association with a visual representation of spoken content |
US8719031B2 (en) | 2011-06-17 | 2014-05-06 | At&T Intellectual Property I, L.P. | Dynamic access to external media content based on speaker content |
US9208471B2 (en) * | 2011-06-24 | 2015-12-08 | Alibaba.Com Limited | Matching users with similar interests |
US8700612B2 (en) * | 2011-08-23 | 2014-04-15 | Contextu, Inc. | Knowledge matrix utilizing systematic contextual links |
US20130080212A1 (en) * | 2011-09-26 | 2013-03-28 | Xerox Corporation | Methods and systems for measuring engagement effectiveness in electronic social media |
JP5958472B2 (en) * | 2011-10-18 | 2016-08-02 | 日本電気株式会社 | Business support device, business support system, business support method, and program |
US8887076B2 (en) * | 2011-11-01 | 2014-11-11 | Aver Informatics Inc. | Software user interface allowing logical expression to be expressed as a flowchart |
US9037590B2 (en) * | 2012-01-23 | 2015-05-19 | Formcept Technologies and Solutions Pvt Ltd | Advanced summarization based on intents |
US9667514B1 (en) | 2012-01-30 | 2017-05-30 | DiscoverReady LLC | Electronic discovery system with statistical sampling |
US10467252B1 (en) | 2012-01-30 | 2019-11-05 | DiscoverReady LLC | Document classification and characterization using human judgment, tiered similarity analysis and language/concept analysis |
US11455350B2 (en) | 2012-02-08 | 2022-09-27 | Thomson Reuters Enterprise Centre Gmbh | System, method, and interfaces for work product management |
US9686212B2 (en) * | 2012-02-08 | 2017-06-20 | Thomson Reuters Global Resources Unlimited Company | Systems, methods and interfaces for using a messaging program across a multiple applications and communications environment |
US8914452B2 (en) | 2012-05-31 | 2014-12-16 | International Business Machines Corporation | Automatically generating a personalized digest of meetings |
GB2504759A (en) * | 2012-08-09 | 2014-02-12 | Navino Evans | Accessing and filtering a database for a category based timeline display |
US10510050B2 (en) * | 2012-09-10 | 2019-12-17 | Private Secretary, Ltd. | Meetings and events coordinating system and method |
US9342580B2 (en) * | 2013-03-15 | 2016-05-17 | FEM, Inc. | Character based media analytics |
US8572097B1 (en) | 2013-03-15 | 2013-10-29 | FEM, Inc. | Media content discovery and character organization techniques |
US9501796B2 (en) * | 2013-09-18 | 2016-11-22 | Chicago Mercantile Exchange Inc. | Dataset intersection determination |
US20150254679A1 (en) * | 2014-03-07 | 2015-09-10 | Genesys Telecommunications Laboratories, Inc. | Vendor relationship management for contact centers |
US11151460B2 (en) | 2014-03-26 | 2021-10-19 | Unanimous A. I., Inc. | Adaptive population optimization for amplifying the intelligence of crowds and swarms |
US12001667B2 (en) | 2014-03-26 | 2024-06-04 | Unanimous A. I., Inc. | Real-time collaborative slider-swarm with deadbands for amplified collective intelligence |
US10416666B2 (en) | 2014-03-26 | 2019-09-17 | Unanimous A. I., Inc. | Methods and systems for collaborative control of a remote vehicle |
US9940006B2 (en) | 2014-03-26 | 2018-04-10 | Unanimous A. I., Inc. | Intuitive interfaces for real-time collaborative intelligence |
US10817158B2 (en) | 2014-03-26 | 2020-10-27 | Unanimous A. I., Inc. | Method and system for a parallel distributed hyper-swarm for amplifying human intelligence |
US10277645B2 (en) | 2014-03-26 | 2019-04-30 | Unanimous A. I., Inc. | Suggestion and background modes for real-time collaborative intelligence systems |
US10817159B2 (en) | 2014-03-26 | 2020-10-27 | Unanimous A. I., Inc. | Non-linear probabilistic wagering for amplified collective intelligence |
US10110664B2 (en) | 2014-03-26 | 2018-10-23 | Unanimous A. I., Inc. | Dynamic systems for optimization of real-time collaborative intelligence |
US10122775B2 (en) | 2014-03-26 | 2018-11-06 | Unanimous A.I., Inc. | Systems and methods for assessment and optimization of real-time collaborative intelligence systems |
US10133460B2 (en) | 2014-03-26 | 2018-11-20 | Unanimous A.I., Inc. | Systems and methods for collaborative synchronous image selection |
US10310802B2 (en) * | 2014-03-26 | 2019-06-04 | Unanimous A. I., Inc. | System and method for moderating real-time closed-loop collaborative decisions on mobile devices |
US11941239B2 (en) | 2014-03-26 | 2024-03-26 | Unanimous A.I., Inc. | System and method for enhanced collaborative forecasting |
US11269502B2 (en) | 2014-03-26 | 2022-03-08 | Unanimous A. I., Inc. | Interactive behavioral polling and machine learning for amplification of group intelligence |
US10551999B2 (en) | 2014-03-26 | 2020-02-04 | Unanimous A.I., Inc. | Multi-phase multi-group selection methods for real-time collaborative intelligence systems |
US10712929B2 (en) | 2014-03-26 | 2020-07-14 | Unanimous A. I., Inc. | Adaptive confidence calibration for real-time swarm intelligence systems |
US10353551B2 (en) | 2014-03-26 | 2019-07-16 | Unanimous A. I., Inc. | Methods and systems for modifying user influence during a collaborative session of real-time collective intelligence system |
EP3123442A4 (en) | 2014-03-26 | 2017-10-04 | Unanimous A.I., Inc. | Methods and systems for real-time closed-loop collaborative intelligence |
US10222961B2 (en) | 2014-03-26 | 2019-03-05 | Unanimous A. I., Inc. | Methods for analyzing decisions made by real-time collective intelligence systems |
US10642471B2 (en) * | 2014-06-25 | 2020-05-05 | Oracle International Corporation | Dual timeline |
JP6244274B2 (en) * | 2014-07-01 | 2017-12-06 | 株式会社日立製作所 | Correlation rule analysis apparatus and correlation rule analysis method |
US20160019316A1 (en) | 2014-07-21 | 2016-01-21 | Splunk Inc. | Wizard for creating a correlation search |
US20160063108A1 (en) * | 2014-08-28 | 2016-03-03 | General Electric Company | Intelligent query for graphic patterns |
US9684694B2 (en) * | 2014-09-23 | 2017-06-20 | International Business Machines Corporation | Identifying and scoring data values |
US9247021B1 (en) | 2014-11-07 | 2016-01-26 | Open Text S.A. | System, method and architecture for a document as a node on a social graph |
US10515082B2 (en) * | 2016-09-14 | 2019-12-24 | Salesforce.Com, Inc. | Identifying frequent item sets |
US10956503B2 (en) | 2016-09-20 | 2021-03-23 | Salesforce.Com, Inc. | Suggesting query items based on frequent item sets |
US10831796B2 (en) * | 2017-01-15 | 2020-11-10 | International Business Machines Corporation | Tone optimization for digital content |
US10170137B2 (en) | 2017-05-18 | 2019-01-01 | International Business Machines Corporation | Voice signal component forecaster |
JP6977565B2 (en) * | 2018-01-04 | 2021-12-08 | 富士通株式会社 | Search result output program, search result output device and search result output method |
US10606954B2 (en) | 2018-02-15 | 2020-03-31 | International Business Machines Corporation | Topic kernelization for real-time conversation data |
US11036751B2 (en) * | 2018-05-07 | 2021-06-15 | Servicenow, Inc. | Advanced insights explorer |
KR102165494B1 (en) * | 2018-12-28 | 2020-10-14 | 네이버 주식회사 | Method, apparatus and computer program for detecting abusing activity in online service |
US11775775B2 (en) * | 2019-05-21 | 2023-10-03 | Salesforce.Com, Inc. | Systems and methods for reading comprehension for a question answering task |
US11880547B2 (en) * | 2019-12-20 | 2024-01-23 | Koninklijke Philips N.V. | Two-dimensional embedding of a hierarchical menu for easy navigation |
US11163942B1 (en) * | 2020-08-04 | 2021-11-02 | International Business Machines Corporation | Supporting document and cross-document post-processing configurations and runtime execution within a single cartridge |
US20220050875A1 (en) * | 2020-08-17 | 2022-02-17 | Business Objects Software Ltd | Time Period Data Comparison |
CN116635844A (en) * | 2020-09-18 | 2023-08-22 | 爱克库尔斯公司 | Dynamic in-transit structuring of unstructured medical documents |
US11402979B1 (en) * | 2021-01-29 | 2022-08-02 | Splunk Inc. | Interactive expandable histogram timeline module for security flagged events |
US11949638B1 (en) | 2023-03-04 | 2024-04-02 | Unanimous A. I., Inc. | Methods and systems for hyperchat conversations among large networked populations with collective intelligence amplification |
Citations (17)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5579466A (en) | 1994-09-01 | 1996-11-26 | Microsoft Corporation | Method and system for editing and formatting data in a dialog window |
WO2000067159A2 (en) | 1999-05-05 | 2000-11-09 | Xerox Corporation | System and method for searching and recommending documents in a collection using shared bookmarks |
WO2001053995A1 (en) | 2000-01-18 | 2001-07-26 | Zantaz.Com | Internet-based archive service for electronic documents |
WO2001057633A1 (en) | 2000-02-07 | 2001-08-09 | John Rudolph | Trust-based cliques marketing tool |
US20010018698A1 (en) | 1997-09-08 | 2001-08-30 | Kanji Uchino | Forum/message board |
WO2001069428A1 (en) | 2000-03-15 | 2001-09-20 | Taalee, Inc. | System and method for creating a semantic web and its applications in browsing, searching, profiling, personalization and advertising |
US20020103792A1 (en) | 2000-02-03 | 2002-08-01 | Arthur Blank | Acumatch cross-matching system |
US6469711B2 (en) | 1996-07-29 | 2002-10-22 | Avid Technology, Inc. | Graphical user interface for a video editing system |
US20020178225A1 (en) | 2001-05-24 | 2002-11-28 | M&G Enterprises Llc | System and method for providing on-line extensions of off-line places and experiences |
US6567830B1 (en) | 1999-02-12 | 2003-05-20 | International Business Machines Corporation | Method, system, and program for displaying added text to an electronic media file |
US6598046B1 (en) | 1998-09-29 | 2003-07-22 | Qwest Communications International Inc. | System and method for retrieving documents responsive to a given user's role and scenario |
US6699125B2 (en) | 2000-07-03 | 2004-03-02 | Yahoo! Inc. | Game server for use in connection with a messenger server |
US7143091B2 (en) | 2002-02-04 | 2006-11-28 | Cataphorn, Inc. | Method and apparatus for sociological data mining |
US7146416B1 (en) | 2000-09-01 | 2006-12-05 | Yahoo! Inc. | Web site activity monitoring system with tracking by categories and terms |
US7188143B2 (en) | 2001-07-06 | 2007-03-06 | Yahoo! Inc. | Messenger-controlled applications in an instant messaging environment |
US7200590B2 (en) | 2001-08-15 | 2007-04-03 | Yahoo! Inc. | Data sharing |
US7412422B2 (en) | 2000-03-23 | 2008-08-12 | Dekel Shiloh | Method and system for securing user identities and creating virtual users to enhance privacy on a communication network |
-
2009
- 2009-04-13 US US12/422,902 patent/US8135711B2/en not_active Expired - Fee Related
Patent Citations (18)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5579466A (en) | 1994-09-01 | 1996-11-26 | Microsoft Corporation | Method and system for editing and formatting data in a dialog window |
US6469711B2 (en) | 1996-07-29 | 2002-10-22 | Avid Technology, Inc. | Graphical user interface for a video editing system |
US20010018698A1 (en) | 1997-09-08 | 2001-08-30 | Kanji Uchino | Forum/message board |
US6598046B1 (en) | 1998-09-29 | 2003-07-22 | Qwest Communications International Inc. | System and method for retrieving documents responsive to a given user's role and scenario |
US6567830B1 (en) | 1999-02-12 | 2003-05-20 | International Business Machines Corporation | Method, system, and program for displaying added text to an electronic media file |
WO2000067159A2 (en) | 1999-05-05 | 2000-11-09 | Xerox Corporation | System and method for searching and recommending documents in a collection using shared bookmarks |
WO2001053995A1 (en) | 2000-01-18 | 2001-07-26 | Zantaz.Com | Internet-based archive service for electronic documents |
US20020103792A1 (en) | 2000-02-03 | 2002-08-01 | Arthur Blank | Acumatch cross-matching system |
WO2001057633A1 (en) | 2000-02-07 | 2001-08-09 | John Rudolph | Trust-based cliques marketing tool |
WO2001069428A1 (en) | 2000-03-15 | 2001-09-20 | Taalee, Inc. | System and method for creating a semantic web and its applications in browsing, searching, profiling, personalization and advertising |
US6311194B1 (en) | 2000-03-15 | 2001-10-30 | Taalee, Inc. | System and method for creating a semantic web and its applications in browsing, searching, profiling, personalization and advertising |
US7412422B2 (en) | 2000-03-23 | 2008-08-12 | Dekel Shiloh | Method and system for securing user identities and creating virtual users to enhance privacy on a communication network |
US6699125B2 (en) | 2000-07-03 | 2004-03-02 | Yahoo! Inc. | Game server for use in connection with a messenger server |
US7146416B1 (en) | 2000-09-01 | 2006-12-05 | Yahoo! Inc. | Web site activity monitoring system with tracking by categories and terms |
US20020178225A1 (en) | 2001-05-24 | 2002-11-28 | M&G Enterprises Llc | System and method for providing on-line extensions of off-line places and experiences |
US7188143B2 (en) | 2001-07-06 | 2007-03-06 | Yahoo! Inc. | Messenger-controlled applications in an instant messaging environment |
US7200590B2 (en) | 2001-08-15 | 2007-04-03 | Yahoo! Inc. | Data sharing |
US7143091B2 (en) | 2002-02-04 | 2006-11-28 | Cataphorn, Inc. | Method and apparatus for sociological data mining |
Non-Patent Citations (7)
Title |
---|
Donath J., et al., Visualizing conversation, Systems Sciences, Jan. 5, 1999, pp. 1-9, IEEE, USA. |
Dunkel, B., et al., Customized metadata for Internet information, Knowledge-Based Intelligent Electronic Systems, KES'97 Proceedings, 1997 1st Int'l Conference on Adelaide, SA Australia, May 21-23, 1997, pp. 508-516. vol. 2, IEEE, New York, USA. |
Katashi Nagao ED. et al, Discussing mining: knowledge discovery from semantically annodated discussion content, New frontries in Artificial Intelligence, Lecture notes in Computer Science; pp. 158-168, vol. 3609, 1900, Springer Berlin Heidelberg, DE, 2007. |
Smith, M.A., et al., Visualization components for persistent conversations, Chi 2001 Conference Proceedings, Mar. 31, 2001, pp. 136-143, IEEE, New York, USA. |
Song, W.W. et al., A semantic similarity approach to electronic document modeling and integration, WEB Information Systems Engineeering, 2000 Proceedings of the 1st international conference on Hong Kong, China, Jun. 19, 2000, pp. 116-124, vol. 1, IEEE Comopu. Soc., Los Alamitos, CA, USA. |
Varlamis, I., et al., Wed document searching using enhanced flyperlink semantics based on xml, Database Engineering & Applications, 2001 Int'l Symposium, Jul. 16-18, 2001. pp. 34-44, IEEE, Piscataway, NJ, USA. |
Warren Sack, Conversation map: a content-based newsgroup browser, Proceedinmgs of the 5th int'l conference on intelligent user interfaces, 2000, pp. 233-240, New Orleans, USA. |
Cited By (23)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140149363A1 (en) * | 2006-10-10 | 2014-05-29 | Adobe Systems Incorporated | Methods and apparatus for automated redaction of content in a document |
US9633063B2 (en) * | 2006-10-10 | 2017-04-25 | Adobe Systems Incorporated | Methods and apparatus for automated redaction of content in a document |
US20090100039A1 (en) * | 2007-10-11 | 2009-04-16 | Oracle International Corp | Extensible mechanism for grouping search results |
US8271493B2 (en) * | 2007-10-11 | 2012-09-18 | Oracle International Corporation | Extensible mechanism for grouping search results |
US9984147B2 (en) | 2008-08-08 | 2018-05-29 | The Research Foundation For The State University Of New York | System and method for probabilistic relational clustering |
US20110074713A1 (en) * | 2009-09-30 | 2011-03-31 | Sony Corporation | Remote operation device, remote operation system, remote operation method and program |
US10795883B2 (en) | 2009-11-13 | 2020-10-06 | Oracle International Corporation | Method and system for enterprise search navigation |
US20110119257A1 (en) * | 2009-11-13 | 2011-05-19 | Oracle International Corporation | Method and System for Enterprise Search Navigation |
US8706717B2 (en) * | 2009-11-13 | 2014-04-22 | Oracle International Corporation | Method and system for enterprise search navigation |
US8751520B1 (en) * | 2010-06-23 | 2014-06-10 | Google Inc. | Query suggestions with high utility |
US8997016B1 (en) * | 2011-01-04 | 2015-03-31 | Intellectual Ventures Fund 79 Llc | Methods, devices, and mediums associated with displaying an electronic pane |
US20140351275A1 (en) * | 2013-05-21 | 2014-11-27 | International Business Machines Corporation | Contributor identification tool |
US9514119B2 (en) * | 2013-05-21 | 2016-12-06 | International Business Machines Corporation | Contributor identification tool |
US10740406B2 (en) | 2013-12-06 | 2020-08-11 | Hewlett-Packard Development Company, L.P. | Matching of an input document to documents in a document collection |
USD800736S1 (en) | 2015-06-16 | 2017-10-24 | Adp Llc | Display screen with animated graphical user interface |
USD786273S1 (en) | 2015-06-16 | 2017-05-09 | Adp, Llc | Display screen with graphical user interface |
US10783496B2 (en) | 2015-06-16 | 2020-09-22 | Adp, Llc | Balanced information system |
US10073926B2 (en) | 2015-08-04 | 2018-09-11 | International Business Machines Corporation | Team analytics context graph generation and augmentation |
US20170039246A1 (en) * | 2015-08-04 | 2017-02-09 | International Business Machines Corporation | Team analytics context graph generation |
US10546020B2 (en) | 2016-02-29 | 2020-01-28 | International Business Machines Corporation | Conversation purpose-based team analytics |
CN107291761A (en) * | 2016-04-05 | 2017-10-24 | 北京优朋普乐科技有限公司 | The matching process and device of a kind of regular expression |
US11151171B2 (en) | 2018-04-27 | 2021-10-19 | International Business Machines Corporation | Accessing faceted information using ontologies |
US11151172B2 (en) | 2018-04-27 | 2021-10-19 | International Business Machines Corporation | Accessing faceted information using ontologies |
Also Published As
Publication number | Publication date |
---|---|
US20090287685A1 (en) | 2009-11-19 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8135711B2 (en) | Method and apparatus for sociological data analysis | |
US7519589B2 (en) | Method and apparatus for sociological data analysis | |
EP1910949A2 (en) | An improved method and apparatus for sociological data analysis | |
Rodriguez et al. | A computational social science perspective on qualitative data exploration: Using topic models for the descriptive analysis of social media data | |
US11409777B2 (en) | Entity-centric knowledge discovery | |
US7143091B2 (en) | Method and apparatus for sociological data mining | |
JP5879260B2 (en) | Method and apparatus for analyzing content of microblog message | |
US8358308B2 (en) | Using visual techniques to manipulate data | |
US7421660B2 (en) | Method and apparatus to visually present discussions for data mining purposes | |
Bartram et al. | Untidy data: The unreasonable effectiveness of tables | |
US20240054104A2 (en) | Dynamic presentation of searchable contextual actions and data | |
US7386439B1 (en) | Data mining by retrieving causally-related documents not individually satisfying search criteria used | |
US20090199090A1 (en) | Method and system for digital file flow management | |
US20130036348A1 (en) | Systems and Methods for Identifying a Standard Document Component in a Community and Generating a Document Containing the Standard Document Component | |
US20130007009A1 (en) | Expertise Tagging and Project Membership Suggestion | |
US20100017431A1 (en) | Methods and Systems for Social Networking | |
Kang et al. | Making sense of archived e‐mail: Exploring the Enron collection with NetLens | |
Franzosi et al. | Qualitative and quantitative research in the humanities and social sciences: how natural language processing (NLP) can help | |
US10776399B1 (en) | Document classification prediction and content analytics using artificial intelligence | |
Wiedmann | Machine learning approaches for event web data extraction | |
Tefera | Implementing an XML Object Identification System on an archive data |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: CATAPHORA, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:CHARNOCK, ELIZABETH;THOMPSON, CURTIS;ROBERTS, STEVEN L;AND OTHERS;REEL/FRAME:022545/0906 Effective date: 20060731 |
|
ZAAA | Notice of allowance and fees due |
Free format text: ORIGINAL CODE: NOA |
|
ZAAB | Notice of allowance mailed |
Free format text: ORIGINAL CODE: MN/=. |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
AS | Assignment |
Owner name: SUNRISE, SERIES 54 OF ALLIED SECURITY TRUST I, CAL Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:CATAPHORA INC.;REEL/FRAME:033710/0173 Effective date: 20140715 |
|
FEPP | Fee payment procedure |
Free format text: PAT HOLDER NO LONGER CLAIMS SMALL ENTITY STATUS, ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: STOL); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
FPAY | Fee payment |
Year of fee payment: 4 |
|
AS | Assignment |
Owner name: ADOBE SYSTEMS INCORPORATED, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SUNRISE, SERIES 54 OF ALLIED SECURITY TRUST I;REEL/FRAME:036890/0269 Effective date: 20151015 |
|
AS | Assignment |
Owner name: ADOBE INC., CALIFORNIA Free format text: CHANGE OF NAME;ASSIGNOR:ADOBE SYSTEMS INCORPORATED;REEL/FRAME:048525/0042 Effective date: 20181008 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 8 |
|
FEPP | Fee payment procedure |
Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
LAPS | Lapse for failure to pay maintenance fees |
Free format text: PATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCH | Information on status: patent discontinuation |
Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362 |
|
FP | Lapsed due to failure to pay maintenance fee |
Effective date: 20240313 |