JP2010231252A - System and method of detecting containment of email content - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a system and a method of detecting containment of email content. <P>SOLUTION: This method includes generating a first set of hash values including hash values corresponding to each of a plurality of character strings of a first email document, generating a second set of hash values including hash values corresponding to each of a plurality of character strings of a second email document, and determining whether the first set of hash values is a subset of the second set of hash values, or not. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to electronic mail systems, and in particular to content inclusion detection in electronic mail documents.

  It is often desirable to efficiently find similar emails that exist in a database. For example, in the case of litigation electronic information disclosure, it is necessary to search an extensive database of e-mails to determine whether the e-mail is important for litigation cases. Searching an extensive database and comparing emails to determine potentially similar ones can be a problematic and tedious process. One approach to compare emails for similarity is to compute hash values from different email content and then compare hash values for identity. Unfortunately, in general, such an approach only identifies email that is an exact copy, and any difference in email generally results in different hash values. Another possible approach is to compare all words of one email with words of another email to determine similarity. However, such techniques are generally very computationally strict.

  Emails are often duplicates because they are forwarded or replied without much text being added. When the first email is repeatedly replied and / or forwarded, it may be desirable to find only the last email in the chain, because the last email is the content of the previous email This is because it often includes all of them. Thus, in an electronic information disclosure scene, it would be more desirable to find the last email in the reply email chain so that the minimum number of emails can be examined without missing any information.

  Disclosed are systems and methods for detecting email content inclusion. In one embodiment, the method includes generating a first set of hash values corresponding to the first e-mail document, wherein the first set of hash values includes a plurality of first e-mail documents. Contains a respective hash value corresponding to each of the strings. The method further includes generating a second set of hash values corresponding to the second e-mail document, the second set of hash values being each of a plurality of character strings of the second e-mail document. Each hash value corresponding to is included. The method finally further includes determining whether the first set of hash values is a subset of the second set of hash values.

  In certain embodiments, the method generates a first Bloom filter representing a first set of hash values corresponding to a first email document, and the method corresponds to a second email document. The method may further include generating a second Bloom filter that represents the second set of hash values and comparing the first Bloom filter to the second Bloom filter. The first and second Bloom filters can be compared by performing a bitwise OR operation. In various embodiments, the method further includes displaying whether the content of the first email document is included in the second email document based on the result of the determining step. .

1 is a block diagram of a computer system including an email database and inclusion detection code. 2 is a flowchart of one embodiment of a method for detecting content inclusion in an email document. 2 shows two exemplary email content. Figure 2 shows two exemplary emails with irrelevant content removed. An exemplary hash is shown. 6 is a flowchart of one embodiment of a method for comparing hash values using a Bloom filter approach. 2 illustrates an exemplary Bloom filter. Fig. 4 illustrates an exemplary bitwise OR comparison of Bloom filters.

  While the invention is susceptible to various modifications and alternative forms, specific embodiments have been shown by way of example and are described in detail herein. However, the drawings and detailed description thereof are not intended to limit the invention to the particular forms disclosed, but rather fall within the spirit and scope of the invention as defined by the claims. It should be understood that variations, equivalents, and alternatives of the above are intended to be included. It should be noted that in this application, the term “may” is used in an acceptable meaning (ie, has a possibility, can be) rather than an essential meaning (ie, must).

  Referring now to FIG. 1, a block diagram of one embodiment of a computer system 100 is shown. Computer system 100 includes a storage subsystem 110 coupled to a processor subsystem 150. The storage subsystem 110 stores an email database 120 and an inclusion detection code 130. The computer system 100 is a consumer such as a personal computer system, desktop computer, laptop or notebook computer, mainframe computer system, handheld computer, workstation, network computer, mobile phone, pager, or personal digital assistant (PDA). It can be any of various types of devices including, but not limited to, a pointing device. The computer system 100 can also be any type of networked peripheral device such as a storage device, a switch, a modem, or a router. Also, although FIG. 1 shows a single computer system 100, the system 100 can also be implemented as two or more computer systems operating simultaneously.

  The processor subsystem 150 illustrates one or more processors that can execute the containment detection code 130. For example, various specific types of processors such as x86 processors, power PC processors, IBM cell processors, or ARM processors can be used.

  Storage subsystem 110, sometimes referred to as "computer readable storage medium", represents a variety of storage media. Storage subsystem 110 may be implemented using any suitable media format and / or storage architecture. For example, the storage subsystem 110 is implemented using a storage medium such as a hard disk storage device, a floppy disk storage device, a removable disk storage, a flash memory, a random access memory, or a semiconductor memory such as a read-only memory. can do. Note that the storage subsystem 110 can run in a single location or can be distributed (eg, in a SAN configuration).

  The email database 120 includes a plurality of email messages, each referred to herein as an email document, associated with one or more email system users. The various email documents in the email database 120 are duplicates of each other, or are substantially the same as other emails in the database (eg, the first email and the corresponding reply email including the first email). Note that content that is similar in nature can be included.

  As will be described in detail below, the containment detection code 130 indicates whether the content of one email document in the database 120 is (or is potentially included) in another email document. Includes instructions executable by the processor subsystem 150 to identify In various embodiments, the email document identified by the inclusion detection code 130 as potentially contained or containing other email content is the user (eg, at least the last email in the chain of reply emails). Email). In certain embodiments, the identified email can be further evaluated. For example, e-mail documents can be analyzed or compared by another code with the same confirmation to determine and / or confirm the extent to which the content of one e-mail is included in another e-mail, and / or electronic Identify email chains. By executing the inclusion detection code 130, it is possible to efficiently filter email documents that do not contain content in other email documents.

  FIG. 2 is a flowchart illustrating various operations that may be performed in accordance with execution of one embodiment of inclusion detection code 130. While the various tasks illustrated in FIG. 2 will be discussed in connection with the exemplary situation illustrated in FIG. 3, FIG. 3 illustrates the contents of two possible email documents 301A and 301B. As illustrated, the e-mail document 301B is a response to the e-mail document 301A. Note that in this example, the emails 301A and 301B include different email headers (eg, sender, destination, and subject portions). Note that the character string “The Fox Was Cunning” that is not included in the e-mail document 011A is included at the end of the e-mail document 301B.

  In step 210, irrelevant email content of the email document being processed is removed or ignored. The irrelevant contents include “From [Name], To [Name], Subject [TITLE], [DATE], [TIME], [NAME] slot”, “Transfer message start”, “----- original Common recurring phrases that can be found in common e-mail documents such as FIG. 4 shows an example resulting from this step, where the header has been removed from the email documents 301A and 301B. In various embodiments, irrelevant email content removed / ignored from each email document during step 210 is a predetermined or preselected word or phrase (eg, commonly common to email documents). ). In other embodiments, irrelevant email content that is removed / ignored can be managed or specified by input from the user. Note that in certain embodiments, step 210 may be omitted.

  Next, in step 220, a first set of hash values is generated for each paragraph of the first email document being processed, and in step 230, a second value is generated for each paragraph of the second email document being processed. A set of hash values is generated. FIG. 5 illustrates an example in which hash values 501A to 501A-E relating to the paragraphs “The quick brown jumped over the lazy dog”, “The dog was sleeping”, and “The Fox was cunning” are generated. In this particular embodiment, the alphabetic position of each character in the paragraph is summed to generate each hash value. For example, the letter “T” is the 20th letter of the alphabet, and the letter “h” is the 8th letter. Accordingly, a hash value of “464” is generated based on the sum of the alphabet positions of the characters in the paragraph “The quick brown jumped over the lazy dog”. Similarly, hash values “189” and “203” are calculated on the basis of the respective paragraphs “The dog was sleeping” and “The Fox was cunning”.

  Note that any variety of other hash functions can be used to calculate the hash value for a particular paragraph. Generally speaking, a “hash function” is any function that maps an input to a number (ie, a hash value). Thus, in various embodiments, specific hash algorithms such as MD5 hash, SHA-1 hash, etc. can be used. In an exemplary embodiment, the input to the hash function may include a value that represents a character, such as a character that forms a paragraph, or an ASCII ordinal value of a character or a letter position of a character within each paragraph. Depending on the embodiment, characters such as punctuation marks and / or numbers may or may not be included as input to the hash function.

  It should also be noted that in certain embodiments, different hash functions can be used to generate multiple hash values for each paragraph. Furthermore, it should be noted that in certain alternative embodiments, a hash value can be calculated for non-paragraph strings, such as a sentence, part of a paragraph, or any other variation for grouping characters. .

  In step 240, the first set of hash values generated in step 220 is compared with the second set of hash values generated in step 230, and the first set of hash values is the second set of hash values. Determine whether to form a subset of values. If the first set is a subset of the second set, the inclusion detection code 130 indicates that the content of the first email is (or may be) included in the second email in step 250A. Display can be made. Conversely, if the first set is not a subset of the second set, the inclusion detection code 130 may not include (or may not include) the content of the first email in the second email in step 250B. Can be displayed. As shown in FIG. 5, hash values 464 and 189 are generated from the respective paragraphs “The quick brown jumped over the lazy dog” and “The dog was sleeping” of the e-mail document 301A. Since these paragraphs of the email document 301A are also included in the content of the email document 301B, the hash values “464” and “189” are also generated for the email document 301B. On the other hand, since “The Fox Was Cunning” is included only in the electronic mail document 301B, the hash value “203” is generated only for the electronic mail document 301B. The set of hash values “464” and “189” corresponding to the e-mail document 301A forms a smaller subset than the set of hash values “464”, “189”, and “203” corresponding to the e-mail document 301B. Therefore, the inclusion detection code 130 can display that the content of the e-mail document 301A is included in the e-mail document 301B. As used herein, if the second set includes each hash value included in the first set, as well as additional hash values generated from paragraphs not included in the first email document, The first set of hash values generated for the first email document forms a small subset of the second email document. In certain embodiments, if the first set is the same as the second set (ie, each set contains the same hash value), the containment detection code 130 provides an indication of content inclusion at step 250A. You can also.

  By repeatedly comparing pairs of different e-mail documents, a chain of reply e-mails can be identified by determining e-mails with content included in other e-mails. If it is determined that one email contains all the other content in the chain, it can be inferred that this email is the last email in the chain. For example, in FIG. 5, email documents 301A and 301B are in the same chain of reply emails, and email document 301B is the last email. In certain embodiments, the inclusion detection code 130 determines that a particular email contains the content of some other email and displays an indication that the particular email is the last in the chain. Can be configured to do.

  Multiple unrelated emails include content that results in a false indication that the content of one email document is included (or potentially included) in another email (eg, a common recurring paragraph) Note that there are occasional cases. Thus, in various embodiments, during step 240, the inclusion detection code 130 can be programmed to ignore specific hash values corresponding to content appearing in multiple unrelated emails.

  FIG. 6 is a flowchart illustrating one particular embodiment of step 240 for determining whether a set of hash values constitutes a small subset of another set. Therefore, the following operations can be performed with the method described above.

  In step 610, the first set of hash values generated in step 220 is reflected in the Bloom filter corresponding to the first email document. Generally speaking, a “bloom filter” is a data structure in the form of a bit vector that represents a set of elements and is used to check whether an element is a member of that set. Initially, an empty Bloom filter can be characterized as a zero bit array. Since the element is added to the Bloom filter, the corresponding representative bit can be set.

  That is, as shown in FIG. 7, by setting the selected bit, the calculated hash value 501A of “464” and the calculated hash value 501B of “189” corresponding to the paragraph from the email document 301A are This is reflected in the filter 701A. In particular, for the specific Bloom filter algorithm shown in this example, bit positions 4 and 6 of the Bloom filter 601A are set based on the numbers forming the calculated hash value “464” and similarly to the hash value “189”. On the other hand, the bits corresponding to positions 1, 8, and 9 are set. In step 620, as shown, the selected bit is similarly set to reflect the calculated hash value generated in step 230 corresponding to the paragraph of the second e-mail document 301B in the Bloom filter 701B. .

  Note that in various embodiments, any of a variety of other Bloom filter algorithms can be used. For example, the size (ie number of bits) of the vector forming the Bloom filter data structure can be much larger than that shown in FIG. 7, and other specific bit positions can be set as directed by the algorithm. Thus, the predetermined hash value can be expressed in the Bloom filter.

  In step 630, the Bloom filters generated in steps 610 and 620 are compared to determine the degree of overlap. As shown in FIG. 6, since the calculated hash values “464” and “189” are shown in both the Bloom filters 701A and 701B, positions 1, 4, 6, 8, and 9 of the Bloom filters 701A and 701B are shown. Are set accordingly. On the other hand, since the hash value “203” is shown only in the Bloom filter 701B, the bits at positions 2, 0, and 3 are not set correspondingly in the Bloom filter 701A.

  In one particular embodiment shown in FIG. 8, a bitwise OR can be performed to compare the Bloom filters of two email documents. In this embodiment, the bit vector 801 is generated from the bitwise OR between the bit vectors of the Bloom filters 701A and 701B and then compared with each of the Bloom filters 701A and 701B. If the resulting bitwise OR bit vector 801 matches either the input Bloom filter 701A or 701B, the inclusion detection code 130 determines that the content of one email is replaced by the content of the other email in step 250A. An indication of being included (or potentially included) can be made. Conversely, if the resulting bitwise OR bit vector 801 does not match either of the Bloom filters 701A or 701B, the containment detection code 130 determines that the content of either email is the content of either email in step 250B. It can be displayed that it is not included (or potentially not included) in the content. In the specific example shown in FIG. 8, since the bit vector 801 matches the Bloom filter 701B, the inclusion detection code 130 displays that the content of the email document 301A is included in the email document 301B. Please note that.

  Although specific embodiments have been described above, these embodiments are intended to limit the scope of the present disclosure even if the specific features are described in only a single embodiment. Not what you want. Examples of features provided in the present disclosure are intended to be illustrative rather than restrictive unless otherwise stated. The above description is intended to cover variations, modifications, and equivalents as may be appreciated by one of ordinary skill in the art having the benefit of this disclosure. The scope of the present invention is disclosed herein (explicitly or implicitly), whether or not alleviating some or all of the problems solved by the various described embodiments. Includes any feature or combination of features, or any generalization. Accordingly, new claims may be made during the examination of this application (or an application claiming priority) for any combination of such features. In particular, with respect to the claims, the features of the dependent claims can be combined with the independent claims, and each independent claim feature can be identified in any suitable manner as specified in the claims. It can be combined not only in the combination.

100 computer system; 110 storage subsystem;
120 email database; 130 inclusion detection code;
150 processor subsystem.

Claims (20)

Generating a first set of hash values corresponding to the first e-mail document and including respective hash values corresponding to each of the plurality of character strings of the first e-mail document;
Generating a second set of hash values corresponding to the second email document and including respective hash values corresponding to each of the plurality of character strings of the second email document;
Determining whether the first set of hash values forms a smaller subset than the second set of hash values;
A method comprising the steps of:   Each of the plurality of character strings of the first e-mail document is a respective paragraph of the first e-mail document, and each of the plurality of character strings of the second e-mail document is the first e-mail document. The method of claim 1, wherein the method is a paragraph of each of the two email documents. Generating a first Bloom filter representing the first set of hash values corresponding to the first email document;
Generating a second Bloom filter representing the second set of hash values corresponding to the second email document;
The method of claim 1, further comprising: comparing the first Bloom filter and the second Bloom filter.   The method of claim 2, wherein the determining includes performing a bitwise OR operation on the first and second Bloom filters.   The method further comprises the step of displaying whether the content of the first e-mail document can be included in the second e-mail document based on the result of the determining step. 4. The method according to 4.   The method further comprises the step of displaying whether the content of the first e-mail document can be included in the second e-mail document based on the result of the determining step. The method according to 1. Generating a first set of hash values corresponding to the first email document and including respective hash values corresponding to each of the plurality of character strings of the first email document;
Generating a second set of hash values corresponding to the second email document and including respective hash values corresponding to each of the plurality of character strings of the second email document;
Determining whether the first set of hash values forms a smaller subset than the second set of hash values;
A computer-readable memory medium having recorded thereon program instructions for executing the procedure on a computer.   Each of the plurality of character strings of the first e-mail document is a respective paragraph of the first e-mail document, and each of the plurality of character strings of the second e-mail document is the first e-mail document. The computer-readable memory medium of claim 7, wherein each is a paragraph of each of two electronic mail documents. The program instruction further comprises:
Generating a first Bloom filter representing the first set of hash values corresponding to the first email document;
Generating a second Bloom filter representing the second set of hash values corresponding to the second email document;
The procedure is computer-executable and determining whether the first set of hash values is a smaller subset than the second set of hash values comprises: 8. The computer readable memory medium of claim 7, comprising a comparison with two Bloom filters.   The program instructions display whether content of the first email document can be included in the second email document based on a comparison of the first and second Bloom filters; The computer-readable memory medium of claim 9, wherein the computer-readable memory medium is executable on a computer.   10. The computer-readable memory medium of claim 9, wherein the program instructions can perform a comparison of the first and second Bloom filters by performing a bitwise OR operation on a computer.   8. The computer-readable memory medium of claim 7, wherein the program instructions are computer-executable to ignore predetermined content of the first and second email documents.   The computer-readable memory medium of claim 12, wherein the predetermined content includes email header information. One or more processors,
Memory,
Generating a first set of hash values corresponding to the first email document and including respective hash values corresponding to each of the plurality of character strings of the first email document;
Generating a second set of hash values corresponding to the second e-mail document and including respective hash values corresponding to each of the plurality of character strings of the second e-mail document;
Determining whether the first set of hash values forms a smaller subset than the second set of hash values;
A memory having recorded program instructions for executing the procedure on a computer;
A system characterized by that.   Each of the plurality of character strings of the first e-mail document is a respective paragraph of the first e-mail document, and each of the plurality of character strings of the second e-mail document is the first e-mail document. 15. The system of claim 14, wherein the system is a paragraph of each of the two email documents. The program instructions are
Generating a first Bloom filter representing the first set of hash values corresponding to the first email document;
Generating a second Bloom filter representing the second set of hash values corresponding to the second email document;
The procedure can be performed on a computer,
The step of determining whether the first set of hash values forms a smaller subset than the second set of hash values compares the first Bloom filter and the second Bloom filter. The system according to claim 14, comprising:   17. The system of claim 16, wherein the program instruction can perform a computer comparison of the first and second Bloom filters by performing a bitwise OR operation.   When the program instruction determines whether the first set of hash values is a smaller subset than the second set of hash values, one of the first or second set or 15. The system of claim 14, wherein ignoring further hash values can be performed by a computer.   Based on the program instructions determining whether the first set of hash values is a smaller subset than the second set of hash values, the second e-mail document is converted to the first e-mail document. 15. The system of claim 14, wherein the system is executable on a computer that identifies the response to the e-mail document.   15. The system of claim 14, wherein one or more of the first and second sets of hash values are generated using an MD5 or SHA-1 hash algorithm.

Images