Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
  • G06F16/33—Querying
  • G06F16/338—Presentation of query results
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
  • G06F16/34—Browsing; Visualisation therefor
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/90—Details of database functions independent of the retrieved data types
  • G06F16/93—Document management systems
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/90—Details of database functions independent of the retrieved data types
  • G06F16/95—Retrieval from the web
  • G06F16/951—Indexing; Web crawling techniques

Definitions

• the present invention relates to systems, methods and interfaces for providing information in response to a computerized search for legal content.
• Primary sources are judicial opinions dealing with the same legal issue. Primary sources are said to be "binding" if the primary source is from a higher court (or the same exact court) than the court currently deciding the legal issue and the higher court is in the "chain of command" of the court currently deciding the issue. For example, the U.S. Supreme Court's opinions are binding on all other courts deciding the same issue. However, a federal district court's opinion from, e.g., New York, is not binding upon a federal district court in Pennsylvania deciding the same issue. The opinion of the New York court is considered persuasive (but not binding) primary authority. Secondary sources comprise other legal content such as law review and other scholarly articles briefs, motions, and administrative decisions.
• West Publishing Company of St. Paul, Minnesota offers the ability for legal professionals to conduct computerized research on over 100 million documents.
• West collects legal content from various sources and makes them available electronically through its Westlaw information-retrieval system.
• Westlaw® is a trademark of West.
• Searchable documents include documents from both primary and secondary sources.
• the West Key NumberTM System which provides classified summaries of legal points, made in judicial opinions, is also searchable (West Key NumberTM is a trademark of Thomson West).
• Headnotes The summaries, known as Headnotes, are classified into more than 90,000 distinct legal categories, and can be used for a variety of purposes, such as evaluating the relevance of legal opinions to particular legal issues.
• Secondary resources such as American Law Reports (ALR®), include about 4,000 in-depth scholarly articles, each teaching about a separate legal issue.
• a method comprising receiving a first signal indicative of a selection of a legal document associated with a set of metadata; based upon the set of metadata, picking a first cluster of legal documents and a second cluster of legal documents, the first cluster of legal documents being associated with a first legal topic and the second cluster of legal documents being associated with a second legal topic; and transmitting a second signal relating to the first cluster of legal documents and the second cluster of legal documents.
• the present invention permits legal professionals to conduct legal research more effectively by providing the legal professional with information regarding clusters of documents associated with a document the legal professional has viewed and/or accessed in some fashion.
• Figure 1 is a diagram of a system corresponding to one embodiment of the invention.
• Figure 2 is a flowchart corresponding to the operation of the system of Figure 1 ;
• Figures 3 A through 31 are screenshots of what a user may see when executing methods in accordance with the invention.
• Figures 4A through 4L relate to the generation of clusters of legal documents.
• a cluster shall mean a set of documents grouped according to a topic that the documents hold in common.
• An example of a cluster is a group of legal documents relating to "search and seizure.”
• a noun phrase is a word group that contains a noun and its modifiers. Examples of noun phrases are "product liability action" and "our favorite restaurant.”
• a segment is a portion of a document that may be defined by the particular topic it addresses. By way of example, a court decision discussing and finding a party liable for fraud and then discussing damages is one document with two segments, namely "fraud” and "damages.”
• the words pick, choose, select, identify, and all respective forms thereof, shall be used interchangeably.
• a document is "associated" with a cluster if it is relevant to the topic of the cluster. Further, a document is a "member” of a cluster if it is both relevant to the topic associated with a cluster and is important in the context of the topic. Still further, a first document is said to be “similar” to a second document if they share a sufficient number of features such as noun phrases and citation history.
• FIG. 1 shows an exemplary online information-retrieval system 100.
• System 100 may include one or more databases 110, one or more servers 120 (only one shown), and one or more access devices 130 (only one shown).
• Databases 110 includes a set of primary databases 112 and a set of second databases 114.
• Primary databases 112 include a case law database 1121 and a statutes databases 1122, which respectively include judicial opinions and statutes from one or more local, state, federal, and/or international jurisdictions.
• Secondary databases 114 include an ALR® database 1141, an AMJUR® database 1142, a West Key NumberTM (KNUM) Classification database 1143, and a law review (LREV) database 1144.
• Other databases may include financial, tax, scientific, and/or health-care information.
• primary and secondary may also connote the order of presentation of search results and not necessarily the authority or credibility of the search results.
• Databases 110 which take the exemplary form of one or more electronic, magnetic, or optical data-storage devices, include or are otherwise associated with respective indices (not shown). Each of the indices includes terms and phrases in association with corresponding document addresses, identifiers, and other conventional information. Databases 110 are coupled or couplable via a wireless or wireline communications network, such as a local-, wide-, private-, or virtual-private network, to server 120.
• a wireless or wireline communications network such as a local-, wide-, private-, or virtual-private network
• Server 120 is generally representative of one or more servers for serving data in the form of web pages or other markup language forms. This may be done with known associated applets, ActiveX controls, remote-invocation objects, or other related software and data structures to service clients of various "thicknesses.” More particularly, server 120 includes a processor module 121 and a memory module 122.
• Processor module 121 includes one or more local or distributed processors, controllers, or virtual machines. In the exemplary embodiment, processor module 121 assumes any convenient or desirable form.
• Memory module 122 takes the exemplary form of one or more electronic, magnetic, or optical data-storage devices. Memory module 122 is comprised of a subscriber database 123, a search module 124, a user-interface module 126, and a cluster module 128.
• Subscriber database 123 includes subscriber-related data for controlling, administering, and managing pay-as-you-go or subscription-based access of databases 110.
• Search module 124 includes one or more search engines and related user- interface components. These search engines receive and process user queries and/or other user activity against one or more of databases 110, including the primary databases 112 and the secondary databases 114.
• the secondary databases may provide, for example, topical treatises, state practice guides, statutes, and/or law review articles to augment searches of case law database.
• User- interface module 126 includes machine readable and/or executable instruction sets for wholly or partly defining web- based user interfaces, such as search interface 1261 and results interface 1262, over a communications link 129 such as a wireless or wireline communications network on one or more accesses devices, such as access device 130.
• a communications link 129 such as a wireless or wireline communications network on one or more accesses devices, such as access device 130.
• Cluster module 128 includes machine readable and/or executable instruction sets. Cluster module 128 interacts, directly and/or indirectly, with the processor 121 and other modules in the memory 122. Cluster module 128 also interacts, directly and/or indirectly, with the databases 110 via communications links 111 and with access device 130 via communications link 129.
• Access device 130 is generally representative of one or more access devices, all of which may simultaneously interact with the server 120.
• access device 130 takes the form of a personal computer, workstation, personal digital assistant, mobile telephone, or any other device capable of providing an effective user interface with a server or database.
• access device 130 includes a processor module 13 lone or more processors (or processing circuits) 131, a memory 132, a display 133, a keyboard 134, and a graphical pointer or selector 135, such as a "mouse.”
• Processor module 131 includes one or more processors, processing circuits, or controllers. In the exemplary embodiment, processor module 131 takes any convenient or desirable form. Coupled to processor module 131 is memory 132.
• Memory 132 stores code (machine -readable or executable instructions) for an operating system 136, a browser 137, and a GUI 138.
• operating system 136 takes the form of a version of the Microsoft® Windows® operating system
• browser 137 takes the form of a version of Microsoft® Internet Explorer®.
• Operating system 136 and browser 137 not only receive inputs from keyboard 134 and selector 135, but also support rendering of GUI 138 on display 133.
• GUI 138 presents data in association with one or more interactive control features (or user-interface elements). (The exemplary embodiment defines one or more portions of interface 138 using applets or other programmatic objects or structures from server 120.)
• graphical user interface 138 defines or provides one or more display regions, such as a query or search region 1381 and a search-results region 1382.
• Query region 1381 is defined in memory and upon rendering includes one or more interactive control features (elements or widgets), such as a query input region 1381A and a query submission button 1381B.
• Search-results region 1382 is also defined in memory and upon rendering includes a first region 1382A, a second region 1382B, and a third region 1382C.
• Region 1382A includes one or more interactive control features, such as features Al, A2, A3 for accessing or retrieving one or more corresponding search result documents from one or more of databases 110 via server 120.
• Region 1382 A in one embodiment, is the region from which a legal professional may select a legal document.
• Regions 1382B and 1382C are, respectively, regions for displaying information relating to the first cluster of legal documents and the second cluster of legal documents. Such information may include respective titles and/or citations for the corresponding documents. For each such documents and/or cluster, this information may be in the form of a hyperlink or other browser-compatible command input that provides access, ultimately, to the documents and/or cluster of documents via server 120 and databases 110.
• FIG. 2 is a flowchart 200 corresponding to operation of the system 100 of Figure 1.
• Flowchart 200 includes blocks 210 through 270 which are arranged and generally described sequentially. However, those skilled in the art realize that other embodiments of the invention may execute two or more blocks in parallel using multiple processors or processor-like devices or a single processor organized as two or more virtual machines or sub processors. Some embodiments also alter the process sequence or provide different functional partitions to achieve analogous results. For example, some embodiments may alter the client-server allocation of functions, such that functions shown and described on the server side are implemented in whole or in part on the client side, and vice versa. Moreover, still other embodiments implement the blocks as two or more interconnected hardware modules with related control and data signals communicated between and through the modules. Thus, the exemplary flowchart of Figure 2 (and elsewhere in this description) applies to software, hardware, and/or firmware implementations.
• FIG. 1 The remaining description in the System Operation section refers to figures 2 through 31 wherein figure 2 outlines the operation of the system 100 and figures 3 A through 31 are various screenshots as seen from the perspective of a user (e.g., legal professional) using a access device 130 to access the WestlawNextTM online information retrieval system.
• a user e.g., legal professional
• FIG. 1 The remaining description in the System Operation section refers to figures 2 through 31 wherein figure 2 outlines the operation of the system 100 and figures 3 A through 31 are various screenshots as seen from the perspective of a user (e.g., legal professional) using a access device 130 to access the WestlawNextTM online information retrieval system.
• a user e.g., legal professional
• the system 100 generates a signal that ultimately causes a search interface to be presented to a user.
• the signal is output from server 120 to access device 130 via communications link 129 and stored in memory 132.
• GUI 138 provides search region 1381 on the access device 130.
• this step assumes that the user operating access device 130 has already successfully logged into the system 100 by supplying an internet-protocol (IP) address for an online information-retrieval system and correct login information (e.g., user identification and password), via the access device 130 and communications link 129, to the system 100.
• IP internet-protocol
• An exemplary search interface screen 300 presented to the user is depicted in Figure 3 A.
• the search interface 300 includes a query input region 310 in which the user of access device 130 may enter a search query by typing text and submitting the query to system 100.
• the system 100 receives the query, also known as a search request, and processes the request.
• the server 120 communicates with at least one database from databases 110 and identifies a set of legal documents in response to the search request.
• the server 120 via the processor 121 and memory 122, generates a signal associated with the set of legal documents identified in response to the search request.
• the signal is transmitted over communications link 129 to access device 130.
• the access device 130 displays a screen 320 to the user based upon this signal.
• Such a screen 320 is depicted in Figure 3B. It should be noted that Figure 3B does not contain information (e.g., titles, words describing, hyperlinks to, etc ..) relating to a first cluster of legal documents and a second cluster of legal documents.
• the system 100 receives another signal generated by the user of access device 130 via communications link 129.
• This signal is indicative of the user accessing a document from the set of legal documents provided in response to the search request. Accessing may be done in a variety of manners including but not limited to the user: (1) viewing the document on the access device 130; (2) printing the document; (3) emailing the document; and (4) setting up an alert with respect to the document.
• the processor 121 and memory 122 begin to process this signal. This is done by identifying a set of metadata associated with the accessed or selected document. This set of metadata is then used to pick a first cluster of legal documents and a second cluster of legal documents as shown in block 250.
• the manner in which clusters are picked is by using a pre-computed set of clusters associated with each document.
• the association process uses a combination of similarity measures between the document and/or document metadata and the cluster and/or cluster metadata. These measures include statistics (such as term-frequency and inverse document-frequency) regarding terms, noun phrases, word pairs, text, citations, associated queries, and other items.
• a signal relating to these clusters is generated and transmitted from server 120 to access device 130 via communications link 129.
• the access device 130 displays a screen 330 to the user based upon this signal. Such a screen 330 is depicted in Figure 3 C.
• the right hand portion 331 of screen 330 is related to the clusters. It should be noted that the right hand portion 331 of screen 330 is analogous to regions 1382B and 1382C of Figure 1. Also, portion 332 of screen 330 is analogous to region 1382 A of Figure 1.
• the server 120 receives and processes this signal by identifying legal documents associated with the topic and sub-topic "Alternative Dispute Resolution/Interstate Commerce Requirement of [the] Federal Arbitration Act.” To process the signal, the server 120 communicates with at least one database from databases 110 and identifies legal documents relevant to the sub-topic (based upon clusters and "sub-clusters"). Next, the server 120, via the processor 121 and memory 122, generates a signal associated with the legal documents and transmits it over communications link 129 to access device 130. The access device 130 displays a screen 340 to the user. Such a screen 340 is depicted in Figure 3D.
• Figures 3E through 31 show another series of screen shots relating to the invention. Essentially they illustrate that another scenario under which signals relating to multiple clusters may be transmitted to an access device. It does not have to be initiated solely in response to a "word" or "text" search (as shown in input region 310 of Figure 3A). For example, Figure 3E begins with a user searching for a document associated with a particular citation, namely 489 U.S. 468, a citation to a Supreme Court case.
• Figures 4A through 4J disclose various algorithms, features and applications for generating and using clusters of legal documents.
• the cluster module 128 of Figure 1 defines and generates a cluster by identifying one or more legal issues among case-law documents, populates the cluster with a rich spectrum of legal documents based upon the cluster's legal issue, summarizes the content represented by the generated cluster, and provides various associations between generated clusters and documents, queries, and folders.
• the description below refers to a Westlaw® system environment, one skilled in the art will appreciate that the disclosed algorithms, features and applications are applicable to other online legal research systems.
• the cluster module 128 implements a bottom-up strategy. For example, in one embodiment, the cluster module 128 identifies the legal issues inside one document, and then merges similar issues together to form clusters for all documents.
• the cluster module 128 identifies legal issues using a Headnotes grouping defined for a case. For example, for cases deemed important on the Westlaw® system, Headnotes (e.g., editorial annotations) are added during the publishing process. Headnotes provide a succinct summary of a legal issue raised in the case and are also associated with one or more Westlaw® Key NumbersTM, described below. An example of a Headnotes grouping with Key NumbersTM is shown in Figure 4A.
• the cluster module 128 identifies major legal issues inside a case.
• the cluster module 128 first computes several features from the Headnotes and then applies an agglomerative clustering algorithm.
• Exemplary similarity features computed by the cluster module 128 include a Key NumbersTM similarity feature, a Headnote text similarity feature, a KeyCite® similarity feature, and a Common Noun Phrase frequency feature.
• the Key NumbersTM similarity feature is based on a Key NumberTM.
• West's Key Number System® is a taxonomy defined on the Westlaw® system that categorizes legal topics into a hierarchical structure.
• the cluster module 128 computes the similarity between Key NumbersTM based on the global co-existence of Key NumbersTM inside cases. In one embodiment, the cluster module 128 determines Key NumberTM topic commonality.
• the Headnote text similarity feature is based on text describing a legal issue. For example, in the Westlaw® system, each Headnote typically includes an amount of text describing a legal issue.
• the cluster module 128 computes the similarity between two Headnotes' text using wordpair features extracted from them. In one embodiment, the cluster module 128 uses a hybrid approach which combines the TF-IDFs (term- frequency- inverse document-frequency) and probabilities of wordpairs.
• the KeyCite® similarity feature is based on relationships between cases.
• KeyCite® data maintains citing and cited relationships between cases (several down to the Headnotes level).
• KeyCite® data includes information concerning the importance/authoritativeness of a case, and information regarding similarity among Headnotes (for example, if two or more Headnotes are co-cited together in several cases, they tend to discuss closely related legal issues).
• U.S. Patent Number 7,529,756 issued on May 5, 2009 entitled "System and Method for Processing Formatted Text Documents in a Database" (filed November 22, 2000 and assigned U.S. Pat. Application Serial No. 09/746,557) and U.S. Pat. Application Serial No.
• the Common Noun Phrase frequency feature is based on a noun phrase (NP) whose head is a noun or a pronoun, optionally accompanied by a set of modifiers.
• NPs typically represent a legal term in a Headnote.
• the cluster module 128 computes the frequency of two common NPs between Headnotes, which provides a measure of how similar Headnotes are at the "concept" level.
• the cluster module 128 uses the NP frequency feature as a supplement to the Headnote text similarity features, since a NP may be considered an n-gram for a particular value of n.
• the cluster module 128 implements an agglomerative clustering algorithm to group similar Headnotes. For example, in one embodiment, the cluster module 128 merges two Headnotes together while maximizing the following equations,
• the cluster module 128 scans through all the Headnote feature vectors, which is one common representation for a set of features used, and identifies two feature vectors which have the maximal ⁇ 2 value.
• the cluster module 128 also computes the value ⁇ i at approximately the same time.
• the cluster module 128 stops the scanning iteration when the value ⁇ i is less than a predefined threshold.
• the cluster module 128 stops the scanning iteration when the value of ⁇ i is less than a predefined threshold.
• the range of the threshold is between 0.0 to 1.0, and preferable, it is set to be 0.45.
• the cluster module 128 avoids setting up the number of clusters for the data set in advance, which many of the known clustering algorithms require.
• the cluster module 128 applies this technique to cases with Headnotes and resulting topics are used in a cluster merging process described below which produces clusters for cases.
• the cluster module 128 is configured to merge similar clusters. For example, legal topics detected in different cases using the before-mentioned techniques may be very similar, i.e., they are concerned with the same or closely related legal issues. By merging similar clusters together, the cluster module 128 partitions the legal space into meaningful clusters.
• the cluster module 128 mergers clusters using a two step process. First, the cluster module 128 performs a candidate selection process.
• the candidate selection process includes generating, training and applying three different CaRE® indices to eligible topics.
• CaRE® stands for Classification and Recommendation Engine.
• CaRE® is described in detail in U.S. Patent No. 7,062,498 which issued on June 13, 2006 entitled “Systems, Methods, and Software for Classifying Text from Judicial Opinions and other Documents" (filed on December 21, 2001 and assigned U.S. Pat. Application Serial No. 10/027,914), U.S. Patent Number 7,580,939 which issued on August 25, 2009 entitled “Systems, Methods, and Software for Classifying Text from Judicial Opinions and other Documents” (filed on August 30, 2005 and assigned U.S. Pat. Application Serial No. 11/215,715), and U.S.
• the cluster module 128 performs the following indexing functions: CaRE® word-pairing indexing, CaRE Key NumbersTM indexing, and CaRE® citation indexing.
• the cluster module 128 associates each topic with a number of Headnote texts.
• the cluster module 128 computes word-pairs of the text and indexes them.
• the cluster module 128 retrieves a list of topics based on the similarities between word- pair profiles.
• the cluster module 128 associates each topic with a list of Key NumbersTM via Headnotes. The cluster module 128 then computes indexed Key NumberTM profiles. The cluster module 128 then retrieves a list of topics based on the commonalities between Key NumberTM profiles.
• the cluster module 128 links each topic to one or more cases, each case is further linked to other cases via KeyCite® information (contain both citing and cited information).
• the cluster module 128 also computes citation profiles that are indexed. The cluster module then retrieves a list of topics based on common citation patterns between citation profiles.
• the cluster module 128 by aggregating the recommendations from the three generated CaRE® indices, the cluster module 128 generates a list of candidates for each of the topics.
• the cluster module 128 determines for each cluster whether the cluster is "similar" to an input topic, and thus should merged with the topic.
• the cluster module 128 For each topic identified, the cluster module 128 generates a query during the Headnotes grouping phrase described previously.
• the query can include noun phrases and Key NumbersTM. An example is shown in connection with Fig. 4B. From the query, along with the associated cases, the cluster module 128 determines several features.
• Exemplary features calculated by the cluster module 128 include Noun Phrases (NPs) similarity - which includes a global maximal score between pair-wise NPs, mean of maximal score between pair-wise NPs, percentage of common NPs, and percentage of common words, Key NumbersTM (KNs) similarity - which includes a Key NumberTM profiles similarity score, percentage of common KNs, and percentage of common KN topics, Co-citation feature - which describes the normalized number of documents cited by both associated seed cases, and Co-click feature, which calculates the normalized number of sessions that have both associated seed cases.
• NPs Noun Phrases
• KNs Key NumbersTM
• Co-citation feature which describes the normalized number of documents cited by both associated seed cases
• Co-click feature which calculates the normalized number of sessions that have both associated seed cases.
• the Co-citation feature describes the normalized number of documents cited by both associated seed cases, and is computed using the following formula: cite(c t n C j ) cite SIm[ Ci, Cj ) — : r v l J J cite (c t U C j ) in which cite(c t n C j ) is the count of other legal documents citing both seed cases cz and cj. Also, cite ⁇ c ⁇ U Cy) is the count of legal documents citing either seed cases cz or cj.
• the co-click feature calculates the normalized number of sessions that have both associated seed cases and is be computed using the following formula:
• the cluster module 128 uses these generated features to train a support vector machine ("SVM") ranker model. SVMs and ranking is well known in the art.
• SVMs and ranking is well known in the art.
• the cluster module 128 In order to provide target data for the training of the model, the cluster module 128 generates a set of "silver" preference grades automatically that measure overlaps between recommended cases from the queries through a search engine process.
• the cluster module 128 In order to provide target data for the training of the model, the cluster module 128 generates a set of "silver” preference grades automatically by measuring the overlaps between retrieved cases using the queries associated with the clusters through a search engine process.
• the search engine is described in detail in U.S. Patent Application No.
• the cluster model 128 By ranking the scores of the candidates using the features via the SVM model, the cluster model 128 generates a cluster by merging selected candidates with a seeding topic based on the ranked scores. A list of clusters can then be produced by exhaustively repeating this process for each of the topics such that one is either merged with other topics or becomes a seeding topic.
• the cluster module 128 generates labels.
• a label displays the "aboutness" of a cluster and reflects a summary of the content inside the cluster.
• the content of a populated cluster can include cases, statutes, regulations, administrative decisions, analytic materials, briefs, expert witness testimony, jury verdict reports, state trial court orders, pleadings, motions and memoranda as well as other legal documents.
• the cases and some of the other documents will also include Headnote texts and Key NumbersTM.
• the catchline of a Key NumberTM is a short description of a defined legal topic, and it is hierarchically structured such that the first portion is often referred to as the Key NumberTM topic, such as "Negligence" in figure 4C, and subsequent portions are often referred to as Key NumberTM sub-topics, while the last portion is often referred to as the leaf level.
• the cluster module 128 generates a hierarchical label structure that includes a topic, optional sub-topic, and a noun phrase from cases.
• the topic and sub-topic parts are derived from Key NumberTM catchlines, which are precise and hierarchically structured phrases describing various legal issues.
• the noun phrase is selected from Headnote texts inside a cluster. Examples of a cluster label is shown below wherein the bold portions represent the topic and sub-topic, and the italic portion is the NP.
• a cluster typically contains a certain number of Key NumbersTM, typically those assigned to the Headnotes contained in the cluster.
• the cluster module 128 computes a frequency of the Key NumbersTM which results in major topics included in the cluster being determined. Once a major topic has been identified, the cluster module 128 traverses the catchlines among Key NumbersTM in the major topic to determine a sub-topic. In one embodiment, the cluster module 128 traverses the catchlines until a divergence is detected based on a majority voting scheme.
• An example of label generation for topics is shown in connection with Figure 4C wherein the label is shown in box 410.
• An example of a majority voting scheme is one where the top n post-divergence sub-topics are considered (where n might be, for example 7) and which selects the sub-topic that is the most frequently occurring within the candidate set.
• the cluster module 128 generates the noun phrase portion by extracting all the Headnote texts inside a cluster. In one embodiment, only those Headnotes in the major Key NumberTM topics are selected by the cluster module 128 for this process.
• NP top scored noun phrase
• the several features include the length of the NP, the term frequency of the composite NP, the term frequency of the NP's terms considered jointly, and the TF-IDF score using normalized TF, As used above, TF stands for term frequency within the given document, DF stands for document frequency within the given collection, and IDF stands for inverse document frequency or the reciprocal of the document frequency.
• weights are determined for this set of features so as to optimize the performance of the label selection process based on empirical evidence from a label grading process. It is also worth noting that for NP scoring and selection purposes an NLP simplified version of the extracted NPs are used (stopped, stemmed, etc.), By contrast, for presentation purposes, a canonical (original) form of the NP is used for user readability.
• the clusters are first ranked by a fitness function that relies on many factors including but not limited to the number of initial cases in the given cluster, and additional features such as the popularity of the cases in the cluster (based on citations and based on user selection), the number of jurisdictions represented, the average age of the cases in the cluster, and the average age of the Key NumbersTM in the cluster.
• a fitness function effectively enables one to rank the clusters by a quality metric.
• the labeling process is applied to the highest quality cluster first, then the next highest, etc....
• the resulting labels are recorded and if a given label has already been assigned to a previously processed cluster, the candidate label is rejected in favor of the next candidate label that has not been previously assigned.
• a semantic representation of each label is recorded, and each candidate label is also assessed for its semantic uniqueness. If a highly semantically similar label has already been assigned, a label can be rejected for a less semantically similar label.
• Processing for this semantic comparison process includes basic natural language processing such as stopping, stemming, term deduping, etc.
• a threshold may also be invoked such that if the core constituent tokens in two labels being compared are 80% similar, they are considered semantically similar, and the candidate will be rejected in favor of the next candidate that is not found to be semantically similar using this threshold.
• the cluster module 1228 applies the search engine process as described in Publication No. U.S. 2008/0033929 Al using the generated query of a cluster.
• the query of a cluster comprises of a number of noun phrases and key numbers.
• the cluster module 128 can tailor the search engine to retrieve the most relevant cases, statutes, regulations, and other documents either online (in real-time) or offline (pre -population).
• FIG. 4D An example workflow of document cluster association is shown in connection with Figure 4D.
• a list of legal topics described in the document is determined by the cluster module 128.
• a list of similar clusters is associated and recommended.
• cluster module 1228 Depending on the metadata available, four different techniques are implemented by the cluster module 128, as illustrated in Figure 4E. For documents with Headnotes defined (cases, some administrative decisions and briefs), the cluster module 128 process operates similarly to the process described in connection with finding legal issues via Headnotes grouping discussed above.
• the cluster module 128 identifies Key NumberTM information from them. The cluster module 128 then groups these Key NumbersTM based on their catchlines such that Key NumbersTM with the most common sub-topics are grouped into one group. An example of Key NumbersTM grouping is shown in connection with Figure 4F.
• the cluster module 128 For documents with citing documents and no Headnotes or NODs, the cluster module 128 incorporates two pieces of information into its method: one is from all the Key NumbersTM of the cited cases and another is from Key NumbersTM suggested by CaRE-KNATM using the document text. KNA stands for Key NumberTM Assignments. The cluster module 128 groups these two sets of Key NumberTM by their topics and then sorts them based on topic popularity. The Key NumbersTM from the cases side with the highest topic popularity that agree with the Key NumbersTM from the CaRE-KNATM side describing the topic level are selected by the cluster module 128 to generate legal topics for the document. Groupings, similar to those shown in Figure 4F, are made by the cluster module 128.
• the summarized document text comprises the first 2,000 characters of the document.
• Those skilled in the art will realize there are other methods for generating summaries of legal documents. Examples of such methods may be found in Schilder, F. and Kondadadi, R, FastSum: Fast and accurate query-based multi-document summarization as contained in the proceedings of the Joint Annual Meeting of the Association for Computational Linguistics and the Human Language Technology Conference (ACL-HLt 2008), pages 205 - 208, Columbus, Ohio, June 2008.
• CaRE-KNATM is a Key Number AssignmentTM service built upon the CaRE® indexing system using the collection of the Key NumbersTM with their corresponding Headnote texts. It can recommend the most relevant Key NumbersTM based on an input query text.
• the cluster module 128 groups these two sets of Key NumbersTM.
• the cluster module 128 For documents with no meta-data but text, the cluster module 128 applies a CaRE- KNA® service to suggest Key NumbersTM based on the text. The Key NumbersTM with the highest topic popularity are then used by the cluster module 128 to perform tasks similar to those shown in Figure 4F to generate legal topics for the document.
• the cluster module 128 associates each document with the pre-defined legal clusters based on its similarity. For example, in one embodiment, the association candidate selection process executed by the cluster module 128 is similar to the candidate selection process for merging clusters described previously. In particular, for all the clusters which can be associated to the topics in legal documents, the cluster module 128 generates the three CaRE® indices based on the word- pair features, Key NumberTM profiles features, and KeyCite® citing/cited profiles features.
• the cluster module 128 For each topic, the three sets of features described previously, namely the word-pair features based on the Headnote text, the Key NumberTM profiles features, and the KeyCite® citing/cited profiles of the seeding document, are calculated by the cluster module 128 and sent to the CaRE® indices. Each CaRE® engine is used to retrieve its independent suggestions, which aggregated later to form a list of candidates to be associated.
• Figure 4G shows an example flowchart of the association candidate selection process.
• the cluster module 128 computes a list of features, as shown and described in connection with Figure 4H.
• the cluster module 128 applies a SVM ranker to these computed features.
• the cluster module 128 selects the top scored candidates as the associated clusters to the topics.
• Figure 41 shows a flowchart illustrating this process.
• the cluster module 128 associates sets of documents stored in folders with a set of document recommendations that address the same legal issue(s) which are relevant to the original document set.
• a "Research Folder" is a place where a user can store together one's documents of interest.
• the research folder can contain various numbers of documents and various document types.
• This folder-based document recommendation method executed by the cluster module 128 identifies common topics (legal issues) among these foldered documents and proceeds to return additional relevant documents that discuss the same topics.
• input to the method is a list of documents, such as cases, statutes, and regulations found in a folder box 480 of Figure 4L.
• the output of the method is a list of additional documents addressing the same distinct legal issues.
• the method involves two steps.
• the cluster module 128 first detects topics and then retrieves the additional documents which share the same legal issues as shown in functional box 481 of Fig. 4L.
• the cluster module 128 uses the relationships among the documents in a folder to find additional relevant documents.
• these relationships identified in the document metadata are exploited.
• These document relationships are quantified by a similarity matrix based on two sources of information. One is the cluster memberships of the documents in the folders. The second is the citation information associated with the documents, citing as well as cited citation information.
• the dimension of the similarity matrix is n x n, where n is, for example, the number of legal cases. Such a matrix could also include other document types such as legal briefs, for example.
• Each entry of the matrix, ay is the similarity score of the document in row i and the document in column j.
• the matrix is sparse (that is, the majority of the entries have 0 values). This property allows for an efficient storage of the entries in a database.
• the matrix is computed offline and the results (entries) are stored in a database (which is part of 482 of Figure 4L.
• a database which is part of 482 of Figure 4L.
• an item-based top-N ranking algorithm is used in functional blocks 483 and 484 to "recommend" the top N documents in response to the documents stored in folders 480.
• one matrix is generated based on document cluster memberships and another matrix is generated based on document citation information. Since both cluster membership and citation information can be used, there can be two scores that exist between two documents.
• the inputs of the recommendation algorithm are the document identifiers of the source documents (from the given folder) along with other useful metadata, for example, the jurisdictions of the documents.
• two recommendation algorithms are in fact run, one based on membership, the other based on citation information., ranking results from both and then combining these results.
• pseudo-code for the recommendation algorithm based upon cluster memberships.
• Such an embodiment may aggregate clusters among those associated to the documents in the folders, and may assign each cluster a combined score (defined as Scorn)-
• the cluster module 128 then sorts these scores in descending order.
• the cluster module 128 computes the combined score Scorn based on topic scores S TP (rank can be implied as well, defined as R TP ), cluster scores S AC (rank can be implied as well, defined as R AC ), and the frequency count (defined as J).
• one folder includes three documents (Doc 1 450, Doc 2 460, and Doc 3 470).
• the first document (Doc 1 450) includes two topics (Topic 1 451 and Topic 2 452), and the second and third documents include one topic (Topic 1 461). Further, each topic is associated with two clusters (CIu N).
• the cluster module 128 computes the combined scores Scorn for other clusters in a similar fashion.
• B is a constant
• i is the z ' th cluster
• R x 1,2, ....
• B is equal to 0.9.
• the cluster module 128 condenses the aggregated clusters into groups such that each group contains highly "similar” clusters, and a representative cluster is selected to for each of the group.
• the cluster module 128 scans through the ordered clusters list and performs a pair-wise similarity comparison between clusters using the information extracted from their queries, namely the NPs and the Key NumbersTM. For clusters with similarity scores above a certain thresholds, the cluster module 128 merges those clusters into a single group. In one implementation using a range of similarity scores from 1 through 5 (with 5 being the most similar), the threshold is 2.7. The cluster module 128 then selects the cluster ranked highest in the ordered list (from the topic detection step described previously with reference, in part, to Figure 4J) to be the representative of the group. The cluster module 128 computes the score of the selected group as the sum of the scores of the clusters in the group. The remainder of the clusters in the group are not visible as output of the algorithm. After the comparison is complete, the cluster module 128 sorts the cluster groups by group score in descending order.
• Example output of the topic consolidation step is shown on the right side of Figure 4K.
• the cluster module 128 grouped clusters CIu 1, Clu2 and Clu5 together, as these clusters were determined to be similar. Also, clusters CIu 3 and CIu 4 were determined to be similar and thus grouped together.
• the cluster module 128 then uses the CIu 1 cluster as being representative of the first group, and cluster CIu 3 as being representative of the second group. Clusters CIu 2, Clu5 and Clu4 are not made available in the output.
• the cluster module 128 also provides a query to clusters association.
• the method of query to cluster association used by the cluster module 128 is similar to the process described in connection with documents having no meta-data. In this case, the query is considered the text.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Databases & Information Systems (AREA)
• Physics & Mathematics (AREA)
• General Engineering & Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Data Mining & Analysis (AREA)
• Computational Linguistics (AREA)
• Business, Economics & Management (AREA)
• General Business, Economics & Management (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
• User Interface Of Digital Computer (AREA)
• Storage Device Security (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=43012722

Family Applications (2)

Family Applications After (1)

Country Status (3)

Families Citing this family (3)

Family Cites Families (8)

• 2010
  • 2010-06-01 CA CA2764319A patent/CA2764319A1/en active Pending
  • 2010-06-01 CA CA2764316A patent/CA2764316C/en active Active
  • 2010-06-01 EP EP10727570A patent/EP2438541A2/de not_active Withdrawn
  • 2010-06-01 WO PCT/US2010/036917 patent/WO2010141480A2/en active Application Filing
  • 2010-06-01 WO PCT/US2010/036913 patent/WO2010141477A2/en active Application Filing
  • 2010-06-01 EP EP10734380A patent/EP2438543A2/de not_active Withdrawn

Non-Patent Citations (2)

Also Published As

Similar Documents

Legal Events