GB2513721A - Computer-implemented systems and methods for comparing and associating objects - Google Patents

Computer-implemented systems and methods for comparing and associating objects Download PDF

Info

Publication number
GB2513721A
GB2513721A GB1404489.5A GB201404489A GB2513721A GB 2513721 A GB2513721 A GB 2513721A GB 201404489 A GB201404489 A GB 201404489A GB 2513721 A GB2513721 A GB 2513721A
Authority
GB
United Kingdom
Prior art keywords
object
objects
slug
plurality
properties
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
GB1404489.5A
Other versions
GB201404489D0 (en
Inventor
Mark Elliot
Allen Chang
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Palantir Technologies Inc
Original Assignee
Palantir Technologies Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority to US201361801297P priority Critical
Priority to US14/099,661 priority patent/US8924388B2/en
Priority to US14/140,415 priority patent/US8924389B2/en
Application filed by Palantir Technologies Inc filed Critical Palantir Technologies Inc
Publication of GB201404489D0 publication Critical patent/GB201404489D0/en
Publication of GB2513721A publication Critical patent/GB2513721A/en
Application status is Withdrawn legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/901Indexing; Data structures therefor; Storage structures
    • G06F16/9014Indexing; Data structures therefor; Storage structures hash tables
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28Databases characterised by their database models, e.g. relational or object models
    • G06F16/284Relational databases
    • G06F16/285Clustering or classification
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/35Clustering; Classification
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/903Querying
    • G06F16/90335Query processing
    • G06F16/90344Query processing by using string matching techniques

Abstract

A method for identifying unique objects within a plurality of objects comprising for each object within the plurality of objects creating a slug for the object, the slug comprising a plurality of properties from the object; and inputting the slug for the object into a counting Bloom filter and for objects in a bin of sixe 1 identifying them as unique and for objects in a bin of size 2 or greater inputting the slugs into a multimap to identify unique objects. This avoids the shortcomings of traditional object comparators by comparing objects in a way that allows for greater computational throughput and acceptable memory consumption without a reduction in comparison accuracy and for dataset sizes that were previously impractical or impossible at acceptable levels of computational throughput

Description

COMPUTER-IMPLEMENTED SYSTEMS AN]) METHODS FOR

COMPARING AND ASSOCIATING OBJECTS

BACKGROUND

Numerous organizations, inchiding industry and government entities, recognize that important conclusions can be drawn if massive data sets can be analyzed to identify patterns of behaviour that suggest dangers to public safety or evidence illegality. These analyses often involve matching data associated with a person or thing of interest with other data associated with the same person or thing to determine that the same person io or thing has been involved in multiple acts that raise safety or criminal concerns.

Yet, the quality of the analytical result arising from use of sophisticated analytical tools can be limited by the quality of data the tool utilizes. For certain types of analyses, an acceptable error rate must be literally or nearly zero for an analytical conclusion drawn i from the data to be sound. Achieving this zero or near-zero error rate for datasets comprising tens or hundreds of millions of records can be problematic. Present data comparison tools are not well suited to solve these issues.

The issues discussed above are particularly acute for analyses involving data related to identifying persons or things for inquiries relating to public safety. For example, analytica' tools for identifying potential safety threats generally do not have an acceptable error rate greater than zero because the cost of mistakenly identifying the presence of a safety threat (i.e., a "false positive") or allowing a safety threat to go undetected (i.e., a "false negative") is unacceptably high. Therefore, tools supporting public safety must correctly relate data associated with persons or things of interest with other data related to the same person or thing.

Some tools exist for accurately comparing data, but they are computationally impractical to use with datasets containing millions of records. For example, one solution to determining whether two particular objects are associated with the same person or thing of interest is to compare each element of one object to a corresponding element in the second object. For example, for objects containing M elements, a first dement in the first object maybe compared to a corresponding first element in the second object, and corresponding comparisons maybe made for each of the remaining M-i elements common to the first and second objects. If the elements within each object are collectivály adequate to uniquely identify the represented person or thing with certainty, and corresponding elements within the first and second objects match, a conclusion may reasonably be drawn that the objects reflect the same person or thing.

As an alternative, each object could be converted (serialized) into a single string reflecting the contents of each &ement to be compared. Thereafter, a string generated from one object could be compared to a string generated from another object as a form of object comparison.

For certain datasets, the above approaches may consume little memory or system o resources, because the objects or their seriafized strings can be stored on disk rather than in main memory. However, the above approaches may quickly become impractical with large or non-trivial datasets. As the number of objects to compare increases, the number of comparisons and thus the processing time of the comparisons increases exponentially; i.e., proportional to n2/2, where n represents the number of objects to be compared. Thus, a comparison of 500 objects using a serialized approach, whose processing time may be approximated as the time to perform 125,000 string comparisons, may be computationafly tractable. However, a comparison of 100 million (looM) records using that approach, whose processing time may be approximated as the time to perform 5 quadrillion (5e15) string comparisons, may be computationally intractable. Additionally, reading strings from disk rather than reading them from memory may add additional processing time.

Another solution for identifying matching objects within a corpus of objects is to store each object in a multimap. This multimap is an associative array that stores multiple values for each key. Importing the objects into the multimap leads to objects with the same element data being stored in a single entry of the multimap. Thus, use of a multimap associates identical objects.

Onc drawback to using a multimap for objcct comparisons is that the multimap is typically stored in main memory, due to algorithmic considerations related to key organization within the multimap, so an object comparator must have sufficient main memory to hold a multimap comprising the entire corpus in memory. Therefore, a multimap solution can be impractical for datasets at or above iooM objects. Similar drawbacks exist to each approach as applied to other object comparison problems, such as efficiently identifying unique objects within a corpus of objects and efficiently comparing a single object to all objects within a corpus of object.

Neither solution is viable for datasets approaching or exceeding looM objects. Yet, object datasets comprising iooM or more objects are not uncommon today. Therefore, the problems described above are quite real and a need exists for improved object comparators.

BRIEF DESCRIVHON OF THE DRAWTNCS

Reference win now be made to the accompanying drawings showing example embodiments of the present application, and in which: FIG.i illustrates a flowchart of an exemplaiy process for comparing a target object to at least some objects in a corpus, consistent with some embodiments of the present

disclosure.

HG.2 illustrates a flowchart of an exemp'ary process for comparing all objects in a corpus to all other objects in the corpus, to determine matches within the corpus, consistent with some embodiments of the present disdosure.

FLG.3 illustrates a flowchart of an exemplary process for comparing all objects in a corpus to all other objects in the corpus, to determine unique objects within the corpus, consistent with some embodiments of the present disclosure.

FIG. 4 illustrates an exemplary computing environment within which embodiments of

the present disclosure can be implemented.

DETATLED DESCRIPTION OF EXEMPLARY EMBODTMENTS

Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. Whenever possible, consistent reference numbers will be used throughout the drawings to refer to the same or like parts.

Embodiments of the present disclosure can avoid the shortcomings of traditional object comparators by providing computer-implemented systems and methods for comparing objects in a way that allows for greater computational throughput and acceptable memory consumption without a reduction in comparison accuracy and for dataset sizes that were previously impractical or impossible at acceptable leváls of computational throughput.

Embodiments of the present disclosure address a class of computational problems related to object comparison. One member of this class inv&ves efficient object comparison of a particu'ar object to a corpus of objects. Another member of this class involves efficient comparison of each object in a corpus to all other objects in the corpus. An additional member of this class involves efficient identification of unique objects within a corpus of objects.

The following detailed description begins with a general overview of object comparison.

Some examples of objects to be compared or analyzed are provided. The description then explains an exemp'ary embodiment that addresses the first class of problem discussed above (i.e., efficiently comparing one object to al objects in a corpus). The description then expands the solution to the first class of problem to address the second class of problem discussed above (i.e., efficient comparison of each object in a corpus to all other objects in the corpus). The detailed description then discloses a solution to the third class of proNem (i.e., efficient identification of unique objects within a corpus of objects). An introduction to objects and an overview of object comparison follows.

Several types of objects exist within the field of computer science. One type of object that is wefl known within the fieki of computer science is an object in the object-oriented sense. Wikipedia describes an object of this type as a set of elements (i.e., data structures) and methods, which are similar to functions. Without necessarily endorsing that rather simplistic description, embodiments implementing the object comparison solutions discussed herein are compatible with comparing objects of this type.

Another type of object within the field of computer science field is a data structure that rcflccts thc properties of a pcrson or thing rclcvant to a particular task or data processing environment. In some embodiments, these properties are reflected by strings. Tn other embodiments, properties maybe reflected by strings, integers, real numbers, times or dates, binary values, structures in the C programming sense, enumerated variables, and/or other forms of data. In some embodiments, properties within either type of object may be converted to strings prior to comparison. In other embodiments, some properties maybe strings or maybe converted to strings while other properties may not be strings and may not be converted to strings. The embodiments of the present disclosure may operate on string or non-string properties.

Moreover, the notion of a "data structure" is very flexible in this context. The term "data structure" can reflect any type of structured data, from information stored in a database (with table columns reflecting elements within an object or data structure and table rows reflecting instances of the object or data structure) to formatted text in a text file (such as data within an XML structure) to data stored within an executing computer program. Accordingly, because a data structure broadly encompasses the types of o structured data described above, objects also broadly encompass these types of structured data. Moreover, the object comparison solutions discussed herein are also compatible with comparing objects of these types.

In some embodiments, effective object comparison involves considering which properties of the objects to be compared are relevant to performing the comparison because the entities (e.g., persons or things) reflected by those objects may have different relevant properties in different environments. For example, an object can store properties of an automobile that may be relevant to a state's motor vehicle department by storing the following information: vehicle identification number (YIN), year of manufacture, make, model, expiration date of the vehicle's registration, and a direct or indirect indication of the person that owns the vehicle.

For automobiles being sold on an auction website such as eBay, however, the relevant properties of an automobile may differ from those relevant to the state's motor vehicle department. For example, a data structure for storing properties of an automobile listed for sale on eBay may include: yIN, year, make, model, odometer reading, condition of the automobile, minimum auction bid, and a direct or indirect indication of the person listing the vehicle for sale. Thus, properties of an entity (e.g., a person or thing) rclcvant to onc cnvironmcnt may diffcr from propcrtics of thc cntity rclcvant to another environment. Accordingly, an object's properties considered during object comparison in one environment may differ from those considered during object comparison in a second environment.

In some embodiments, effective data comparison may also involve considering which properties tend to distinguish an entity (e.g., a person or thing) from other instances of the entity. For example, a YIN for an automobile should by design be unique to that automobile. However, occasional situations may arise where a YIN is not unique to a particular automobile. Such situations may arise from intentional errors or accidental errors. An examp'e of an intentional error is attempting fraudifient registration of a stolen vehicle under an assumed YIN. An example of an accidental error occurs when a smog check worker incorrectly enters a YIN into a computer at a smog check station, which leads to a smog check record with an incorrect YiN subsequently being communicated to a state database. Data errors exist in real world data processing environments, so some embodiments of the present disclosure minimize or eliminate errors by identifying objects through a combination of several object properties rather Jo than identifying objects through use of a single object property.

In some embodiments, one or more identifying properties of an object are extracted from the object and stored in a data structure. This data structure is referred to as a "slug"; it contains information that may be sufficient to uniquely identify an entity (e.g., a person or thing) with some degree of information redundancy to allow for detecting errors in the properties within the slug. Tn some embodiments, the slug comprises a concatenation of strings separated by a delimiter character. Tn some embodiments, the delimiter character is a NULL character while in other embodiments the delimiter character may be a character not otherwise present in the concatenated string. In some embodiments, the concatenated strings may be delimited by a delimiter string (e.g.,"-- ") rather than a delimiter character. In embodiments employing a delimiter string, the delimiter string may be any string that is not otherwise present in the strings that were concatenated. In other embodiments, the slug comprises a data structure such as an object, array, structure, or associative array.

For example, in one embodiment, slug for an automobile may contain properties reflecting a YIN, make, model, and year for the automobile. Inclusion of make, model, and year properties for the automobile within the slug provides a capability for dctccting crrors in tim YEN propcrty bccausc thc YIN property is not thc only objcct property being compared. For slugs associated with two automobiles to match in the presence of an error in the YIN property of one automobile object, an automobile object with the same YIN property as the erroneous YIN must a'so have the same make, model, and year properties.

The odds of this coincidental match of multipk properties between two or more objects may be fleetingly low. Therefore, inclusion of some degree of information redundancy should avoid or at least substantially reduce erroneous object comparison matches relative to object comparisons only comparing a single property between objects notwithstanding that the single propertywas intended to uniquely identify its corresponding entity (e.g., person or thing).

Exemplary embodiments will now be described that solve the first prolMem discussed above, i.e., efficiently comparing a particu'ar object (hereinafter a "target object") to all objects in a corpus. The disclosed embodiments utilize a Bloom filter to identify slugs associated with objects in the corpus that do not match the slug for the target object.

This quick recognition is performed by discarding slugs that are associated with a Jo different bin in the Bloom filter than the bin associated with the slug for the target object.

Bloom filters have the property that two slugs falling into different bins within the Bloom filter are certain to have different properties and thus reflect different objects.

Therefore, if the slug for the target object does not fall into the same bin as the slug for a particuthr object in the corpus, the target object does not match the particular object in the corpus and may thus be removed from future consideration in such embodiments.

FIG.i illustrates a flowchart of an exemplary process 100 for comparing a target object to at kast some objects in a corpus, consistent with some embodiments of the present disclosure. In some embodiments, the target object to be compared to at least some objects in the corpus is a member of the corpus. In these embodiments, a comparison between the target object and all other objects in the corpus is performed. In other embodiments, the object to be compared to at least some objects in the corpus is not a member of the corpus. In these other embodiments, a comparison between the target object and all objects in the corpus is performed.

As illustrated, in stcp 102, a Bloom filtcr is sizcd and crcatcd with considcration for thc error rate that win result for the corpus size that is being processed. For example, increasing the number of bins in a Bloom filter may tend to decrease the error rate for a specific corpus size while reducing the number of bins in a Bloom filter may tend to increase the error rate for a specific corpus size. Techniques for sizing a Bloom filter to achieve a target error rate for a specific corpus size are well known in the art, so these s techniques are not discussed herein.

In step 104, a slug for the target object (i.e., the object against which all objects in the corpus will be compared) is generated. Considerations for selecting which properties of an object to include in a slug were discussed above. Tn step 106, a Bloom filter bin corresponding to the slug for the target object is determined. Tn some embodiments, a Bloom filter bin for a slug may be determined by inputting the slug to a Bloom filter and directing the Bloom filter to disclose the bin into which the slug was added.

In other embodiments, a Bloom filter bin for a slug may be determined by presenting the slug as a input to a software function associated with the Bloom filter without Jo storing the slug in the Bloom filter. In additional embodiments, a bin for a shig maybe determined by inputting the slug into a software function reflecting a bin selection algorithm for a Bloom filter in the absence of using an actual and/or complete Bloom filter and receiving the Bloom filter bin as an output of that software function. In other embodiments, other approaches to yielding a Bloom filter bin from a slug may be utilized. These approaches for identifying a Bloom filter bin for a slug, consistent with the embodiments discussed above, are coflectively referred to in steps 106, 108. The determined Bloom filter bin will be utilized to identify slug comparison matches, some of which may be "false positives", using the Bloom filter as discussed below.

In step io8, a slug for each object in the corpus is generated. In step 110, a Bloom filter bin for each object in the corpus is determined. In some embodiments, a Bloom filter bin for an object may be determined by inputting the object's slug into the Bloom filter and directing the Bloom filter to disclose the bin into which the slug was added.

After completion of step no, slugs corresponding to the bin identified in step io8 reflect matches with the slug for the target object. Some of these matches, however, may be false positive matches rather than true matches. Therefore, steps 112 and 114 filter out the false positive matches through use of a multimap.

Tn step 112, for each slug corresponding to an object in the corpus whose bin in the Bloom filter is the same bin as the slug for the target object, the s'ug corresponding to an object in the corpus and its corresponding object in the corpus is added to a multimap. When adding the slug and its corresponding object to the multimap, the slug represents the key to the multimap and the object in the corpus represents the value to the multimap. This multimap will be utilized to remove false positives from processing. tn step 114, the process concludes by selecting the true positive matches identified in the multimap. These non-false positive matches can be retrieved from the multimap by reading data from the multimap with the slug for the target object as a key.

In some embodiments, process 100 may be distributed across multiple processors. For example, a Bloom filter may exist on each of several processors and steps 102 through 114 can be executed on each of the several processors. The corpus of objects maybe distributed among the various processors so that ali objects are processed by one processor, but no object is processed by more than one processor. In such o embodiments, each of the multiple processors outputs a portion of the objects in the corpus that match the target object.

Exemplary embodiments will now be described that solve the second problem discussed above, i.e., efficiently comparing all objects to all objects in a corpus. These embodiments utilize a counting moom filter to quickly identify slugs associated with objects in the corpus that do not match the slug for the target object. Counting Bloom filters are wefl known in the art, so their structure and construction are not discussed herein.

In particular, if a bin in the counting Bloom filter has a value of zero or one after slugs for all of the objects in the corpus have been input to the Bloom filter, no object whose slug is associated with that bin could match another slug, so these slugs are removed from further consideration. These slugs can be removed because those skilled in the art will recognize that Bloom filters can have false positives but they cannot have false negatives. Therefore, a counting Bloom filter bin whose count is less than two reflects an accurate determination that no match exists between slugs associated with that bin because any match would create a count of at least two. However, false positive may exist among objects whose slugs are associated with the same Bloom filter bin, so false positivcs may bc rcmovcd through additional proccssing, as discusscd bclow.

FLC-.2 iflustrates a flowchart of an exemplary process 200 for comparing all objects in a corpus to all other objects in the corpus, to determine matches within the corpus, consistent with some embodiments of the present disclosure. As illustrated, in step 202, a counting Bloom filter is sized and created with consideration for the error rate that will result for the corpus size that is being processed. For example, increasing the number of bins in a counting Bloom filter may tend to decrease the error rate for a -10-specific corpus size while reducing the number of bins in a counting Bloom filter may tend to increase the error rate for a specific corpus size. Techniques for sizing a counting Bloom filter to achieve a target error rate for a specific corpus size are well known in the art, so these techniques are not discussed herein.

In some embodiments, the counting Bloom fitter may comprise an N-bit counter and these counters maybe implemented as two-bit counters (i.e., N=2). In other embodiments, these counters may be one-bit counters or counters of more than two bits. In additional embodiments, these counters are saturation counters; i.e., these o counters will count up to a maximum value and then not exceed that value.

In step 204, a slug for each object in the corpus is generated. Instep 206, each slug is input to the counting Bloom filter, which causes a counter in a bin corresponding to a slug to be incremented. After completion of step 206, bins whose counters have a value greater than one reflect one or more matching slugs. Some of these matches, however, may be false positive matches rather than true matches. Therefore, steps 208 and 210 filter out the false positive matches through use of a multimap.

In step 208, for slugs associated with bins in the counting Bloom filter whose counters have a value greater than 1, the slug and its associated object are added to a multimap.

When adding the slug and its corresponding object to the multimap, the s'ug represents the key to the muttimap and the object in the corpus represents the value to the muttimap. This multimap will be utilized to remove false positives from processing. In step 210, the process 200 concludes by outputting a value for any key in the multimap that has two or more va'ues. The outputted values reflect objects whose slugs matched slugs of at least one other object in the corpus. Thus, the objects outputted identify objects whose selected properties, as reflected in an object's slug, unambiguously match at least one other object in the corpus.

Tn some embodiments, process 200 maybe distributed across multiple processors. For examp'e, a counting Bloom filter may exist on each of severa' processors and steps 202, 204, and 206 can be executed on each of the several processors. The corpus of objects may be distributed among the various processors so that all objects are processed by one processor, but no object is processed by more than one processor. In such embodiments, prior to executing step 208, counters for each bin in the counting Bloom filter are summed together with counters for the same bin in counting Bloom filters on -11 -other processors. Thereafter, process 200 continues by executing steps 208 and 210 on a sing'e processor.

Exemplary embodiments will now be described that sohTe the third problem discussed above, i.e., efficiently identifying unique objects in a corpus. These embodiments utilize a counting Bloom filter and a multimap to quickly identify unique objects. Upon inputting slugs for all objects in the corpus into the counting Bloom filter, any bin with a count value of one reflects a unique object because Bloom filters do not generate false negatives. Additional'y, to the extent that bins have count values of two or more, those o count values coffid reflect false positives. Therefore, a multimap allows a determination of whether the matches reflected in the count values were false or tnie positives.

FIG.3 illustrates a flowchart of an exemplary process 300 for comparing all objects in a corpus to all other objects in the corpus, to determine unique objects within the corpus, consistent with some embodiments of the present disclosure. As illustrated, in step 302, a counting Bloom filter is sized and created with consideration for the error rate that will result for the corpus size that is being processed. For example, increasing the number of bins in a counting Bloom filter may tend to decrease the error rate for a specific corpus size while reducing the number of bins in a counting Bloom filter may tend to increase the error rate for a specific corpus size. Techniques for sizing a counting Bloom filter to achieve a target error rate for a specific corpus size are well known so these techniques are not discussed herein.

In some embodiments, the counting Bloom filter may comprise an N-bit counter and these counters may be implemented as two-bit counters (i.e., N=2). In other embodiments, these counters may be one-bit counters or counters of more than two bits. In additional embodiments, these counters are saturation counters; i.e., these counters will count up to a maximum value and then not exceed that value.

Tn step 304, a slug for each object in the corpus is generated. In step 306, each slug is input to the counting Bloom filter, which causes a counter in a bin corresponding to the slug to be incremented. As previously discussed, after slugs for afi objects in the corpus have been input to the counting Bloom filter, any bin with a count value of one reflects a unique object within the corpus because the counting Bloom ifiter does not generate false negatives. Therefore, in step 308, for each slug whose counter in the counting -12 -Bloom filter is one, the slugs corresponding object is output as a unique object within the corpus.

After completion of step 308, bins whose counters have a value greater than one reflect one or more matching slugs; i.e., slugs that are not unique. Some of these matches, however, may be false positive matches rather than true matches dne to the nature of Bloom filters, as discussed above. Therefore, steps 310 and 312 filter out the false positive matches through use of a multimap.

o Steps 310 and 312 determine whether the counting Bloom filter is masking the existence of other unique objects because the moom filter allows for false positives. In step 310, for each slug whose associated bin has a counter value greater than one, the slug is input as a key to a multimap and the object corresponding to the slug is input as a value for that key. In step 312, the process terminates after outputting each value in the multimap for keys that have only one value. Unique objects within the corpus are reflected by the collection of objects output from step 308 and the coflection of objects output by step 312 because the former reflects objects whose slugs were the only slug in a counting Bloom filter's bin and were therefore unique among slugs associated with objects in the corpus while the latter reflects slugs that were false positives within the counting Bloom filter but were disambiguated by the multimap.

In some embodiments, process 300 maybe distributed across multiple processors. For example, a counting Bloom filter may exist on each of several processors and steps 302, 304, and 306 can be executed on each of several processors. The corpus of objects may be distributed among the various processors so that all objects are processed by one processor, but no object is processed by more than one processor. In such embodiments, prior to executing step 308, counters for each bin in the counting Bloom filter are summed together with counters for the same bin in counting Bloom filters on othcr proccssors. Thcrcaftcr, proccss 300 continucs by cxccuting stcps 308,310, and 312 on a single processor.

FIG. 4 illustrates an exemplary computing environment within which the embodiments of the present disclosure can be imp'emented.

Computer system 400 includes a bus 402 or other communication mechanism for communicating information, and a hardware processor 404 coupled with bus 402 for -13 -processing information, in some embodiments, hardware processor 404 can be, for example, a general-purpose microprocessor or it can be a reduced instruction set microprocessor.

Computer system 400 also includes a main memory 406, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 402 for storing information and instructions to be executed by processor 404. Main memory 406 also can be used for storing temporary variables or other intermediate information during execution of instructions by processor 404. Such instructions, when stored in non-o transitory storage media accessible to processor 404, render computer system 400 into a speciahpurpose machine that is customized to perform the operations specified in the instructions.

In some embodiments, computer system 400 further includes a read only memory (ROM) 408 or other static storage device coupled to bus 402 for storing static information and instructions for processor 404. A storage device 410, such as a magnetic disk or optica' disk, is provided and coupled to bus 402 for storing information and instructions.

Computer system 400 can be coupled via bus 402 to a display 412, such as a cathode ray tube (CRT) or LCD panel, for displaying information to a computer user. An input device 414, including alphanumeric and other keys, is coupled to bus 402 for communicating information and command selections to processor 404. Another type of user input device is cursor control 416, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor 404 and for controlling cursor movement on display 412. The input device typically has degrees of freedom in two axes, a first axis (for example, x) and a second axis (for example, y), that allows the device to specify positions in a plane.

Computer system 400 can implement the processes and techniques described herein using customized hard-wired logic, one or more ASTCs or FPGAs, firmware and/or program logic which in combination with the computer system causes or programs computer system 400 to be a special-purpose machine. In some embodiments, the processes and techniques herein are performed by computer system 400 in response to processor 404 executing one or more sequences of one or more instructions contained in main memory 406. Such instructions can be read into main memory 406 from another storage medium, such as storage device 410. Execution of the sequences of instructions contained in main memory 406 causes processor 404 to perform the process steps described herein. Tn other embodiments, hard-wired circuitry can be used in place of or in combination with software instructions.

The term "storage media" as used herein refers to any non-transitory media that store data and/or instructions that cause a machine to operate in a specific manner. Such storage media can comprise non-volatile media and/or volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as storage device 410.

o Volatile media includes dynamic memory, such as main memory 406. Common forms of storage media include, for example, a floppy disk, a flexible disk, hard disk, solid state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge.

Storage media is distinct from but can be used in conjunction with transmission media.

Transmission media participates in transferring information between storage media.

For example, transmission media includes coaxial cables, copper wire and fibre optics, including the wires that comprise bus 402. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications.

Various forms of media can be involved in carrying one or more sequences of one or more instructions to processor 404 for execution. For example, the instructions can initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 400 can rcccivc thc data on thc tclcphonc linc and usc an infra-rcd transmittcr to convert the data to an infra-red signal. An infra-red detector can receive the data carried in the infra-red signal and appropriate circuitry can place the data on bus 402.

Bus 402 carries the data to main memory 406, from which processor 404 retrieves and executes the instructions. The instructions received by main memory 406 can optionally be stored on storage device 410 either before or after execution by processor 404.

-15 -Computer system 400 a'so includes a communication interface 418 coupled to bus 402.

Communication interface 418 provides a two-way data communication coupling to a network Unk 420 that is connected to a loca' network 422. For example, communication interface 418 can be an integrated services digital network (TSDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface 418 can be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links can also be implemented. In any such implementation, communication interface 418 sends and jo receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

Network link 420 typically provides data communication through one or more networks to other data devices. For example, network link 420 can provide a connection through ocal network 422 to a host computer 424 or to data equipment operated by an Tnternet Service Provider (TSP) 426. TSP 426 in turn provides data communication services through the world wide packet data communication network now commonly referred to as the "Internet" 428. Local network 422 and Tnternet 428 both use electrical, electromagnetic or optical signals that carry digital data streams.

The signals through the various networks and the signals on network link 420 and through communication interface 418, which carry the digital data to and from computer system 400, are example forms of transmission media.

Computer system 400 can send messages and receive data, including program code, through the network(s), network link 420 and communication interface 418. Tn the Tnternet example, a server 430 might transmit a requested code for an application program through Tnternet 428, ISP 426, 1oca network 422 and communication interface 418. The received code can be executed by processor 404 as it is received, and/or stored in storage device 410, or other non-volatile storage for later execution.

Claims (20)

  1. Claims 1. A method for identifying unique objects within a plurality of objects, each object comprising a first plurafity of properties, each property comprising data reflecting a characteristic of an entity represented by the object, the method comprising the following operations performed by one or more processors: executing, for each object within the plurality of objects, the following: creating a slug for the object, the slug comprising a second plurality of properties from the object; and inputting the slug for the object into a counting Bloom filter; identifying for each created slug whose corresponding bin within the counting Bloom filter has a count value equal to 1, the object associated with the slug as unique within the plurality of objects; inputting, using at least one processor, for each created slug, the slug and its corresponding object into a multimap, ifa bin within the counting ffloom filter corresponding to the slug has a count value greater than 1, wherein the slug is a key to the multimap and the object is a value to the multimap; and identifying for each multimap key with one value, the object associated with the slug stored as the key as unique within the plurality of objects.
  2. 2. The method of claim 1, further comprising: sizing the counting Bloom filter for a predetermined error rate and number of objects within the plurality of objects.
  3. 3. The method of claim 2, further comprising: reading the plurality of objects from at least one database.
  4. 4. The method of claim 3, wherein each entry in the counting Bloom filter compriscs a 2-bit countcr.
  5. 5. The method of daim 4, wherein each 2-bit counter is a saturation counter.
  6. 6. The method of élaim 5, wherein the delimiter comprises a character that is not otherwise present in the strings that were concatenated. -17-
  7. 7. The method of claim 5 or claim 6, wherein the delimiter comprises a sequence of two or more characters and the sequence of two or more characters is not present in any of the two or more strings that were concatenated.
  8. 8. The method of any of claims j to 7, wherein the number of properties in the first plurality of properties equals the number of properties in the second plurality of properties.
  9. 9. The method of any of claims 5 to 7, wherein the number of properties in the first Jo plurality of properties is greater than the number of properties in the second phirality of properties.
  10. 10. A system for identifying unique objects within a plurality of objects, each object comprising a first plurality of properties, each property comprising data reflecting a characteristic of an entity represented by the object, the system comprising: a memory device that stores a set of instructions; and at least one processor that executes the set of instructions to perform the following operations: execute, for each object within the plurality of objects, the following: create a slug for the object, the slug comprising a second plurality of properties from the object; and input the slug for the object into a counting Bloom filter; identify, for each created slug whose corresponding bin within the counting Bloom filter has a count value equal to 1, the object associated with the slug as unique within the plurality of objects; input, for each created slug, the slug and its corresponding object into a multimap, if a bin within the counting Bloom filter corresponding to the slug has a count value greater than 1, wherein the slug is a key to the multimap and thc object is a value to the multimap; and identify, for each muhimap key with one value, the object associated with the slug stored as the key as unique within the plurality of objects.
  11. ii. The system of claim 10, wherein the at least one processor further executes the set of instructions to perform the following operation: sizing the counting Bloom filter for a predetermined error rate and number of objects within the plurality of objects.-18 -
  12. 12. The system of claim ii, wherein the at least one processor further executes the set of instructions to perform the following operation: reading the plurality of objects from at least one database.
  13. 13. The system of claim 12, wherein each entry in the counting Bloom filter comprises a 2-bit counter.
  14. 14. The system of claim 13, wherein each 2-bit counter is a saturation counter.
  15. 15. The system of claim 14, wherein the s'ug comprises a concatenation of two or more strings separated by a delimiter between each concatenated string.
  16. 16. The system of claim 15, wherein the delimiter comprises a character that is not otherwise present in the strings that were concatenated.
  17. 17. The system of claim i or claim 16, wherein the delimiter comprises a sequence of two or more characters and the sequence of two or more characters is not present in any of the two or more strings that were concatenated.
  18. 18. The system of any of claims 15 to 17, wherein the number of properties in the first plurality of properties equals the number of properties in the second plurality of properties.
  19. 19. The system of any of claims 15 to 17, wherein the number of properties in the first plurality of properties is greater than the number of properties in the second plurality of properties.
  20. 20. A computer program comprising machinc rcadablc instructions that when executed by computing apparatus cause it to perform the method of any of claims ito 9
GB1404489.5A 2013-03-15 2014-03-13 Computer-implemented systems and methods for comparing and associating objects Withdrawn GB2513721A (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
US201361801297P true 2013-03-15 2013-03-15
US14/099,661 US8924388B2 (en) 2013-03-15 2013-12-06 Computer-implemented systems and methods for comparing and associating objects
US14/140,415 US8924389B2 (en) 2013-03-15 2013-12-24 Computer-implemented systems and methods for comparing and associating objects

Publications (2)

Publication Number Publication Date
GB201404489D0 GB201404489D0 (en) 2014-04-30
GB2513721A true GB2513721A (en) 2014-11-05

Family

ID=50634708

Family Applications (1)

Application Number Title Priority Date Filing Date
GB1404489.5A Withdrawn GB2513721A (en) 2013-03-15 2014-03-13 Computer-implemented systems and methods for comparing and associating objects

Country Status (2)

Country Link
DE (1) DE102014204834A1 (en)
GB (1) GB2513721A (en)

Cited By (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9286373B2 (en) 2013-03-15 2016-03-15 Palantir Technologies Inc. Computer-implemented systems and methods for comparing and associating objects
US9348499B2 (en) 2008-09-15 2016-05-24 Palantir Technologies, Inc. Sharing objects that rely on local resources with outside servers
US9392008B1 (en) 2015-07-23 2016-07-12 Palantir Technologies Inc. Systems and methods for identifying information related to payment card breaches
US9483546B2 (en) 2014-12-15 2016-11-01 Palantir Technologies Inc. System and method for associating related records to common entities across multiple lists
US9495353B2 (en) 2013-03-15 2016-11-15 Palantir Technologies Inc. Method and system for generating a parser and parsing complex data
US9501552B2 (en) 2007-10-18 2016-11-22 Palantir Technologies, Inc. Resolving database entity information
US9514414B1 (en) 2015-12-11 2016-12-06 Palantir Technologies Inc. Systems and methods for identifying and categorizing electronic documents through machine learning
US9715518B2 (en) 2012-01-23 2017-07-25 Palantir Technologies, Inc. Cross-ACL multi-master replication
US9760556B1 (en) 2015-12-11 2017-09-12 Palantir Technologies Inc. Systems and methods for annotating and linking electronic documents
US9852205B2 (en) 2013-03-15 2017-12-26 Palantir Technologies Inc. Time-sensitive cube
US9880987B2 (en) 2011-08-25 2018-01-30 Palantir Technologies, Inc. System and method for parameterizing documents for automatic workflow generation
US9898335B1 (en) 2012-10-22 2018-02-20 Palantir Technologies Inc. System and method for batch evaluation programs
US9984428B2 (en) 2015-09-04 2018-05-29 Palantir Technologies Inc. Systems and methods for structuring data from unstructured electronic data files
US9996229B2 (en) 2013-10-03 2018-06-12 Palantir Technologies Inc. Systems and methods for analyzing performance of an entity
US10061828B2 (en) 2006-11-20 2018-08-28 Palantir Technologies, Inc. Cross-ontology multi-master replication
US10103953B1 (en) 2015-05-12 2018-10-16 Palantir Technologies Inc. Methods and systems for analyzing entity performance
US10127289B2 (en) 2015-08-19 2018-11-13 Palantir Technologies Inc. Systems and methods for automatic clustering and canonical designation of related data in various data structures
US10133588B1 (en) 2016-10-20 2018-11-20 Palantir Technologies Inc. Transforming instructions for collaborative updates
US10140664B2 (en) 2013-03-14 2018-11-27 Palantir Technologies Inc. Resolving similar entities from a transaction database
US10180977B2 (en) 2014-03-18 2019-01-15 Palantir Technologies Inc. Determining and extracting changed data from a data source
US10198515B1 (en) 2013-12-10 2019-02-05 Palantir Technologies Inc. System and method for aggregating data from a plurality of data sources
US10235533B1 (en) 2017-12-01 2019-03-19 Palantir Technologies Inc. Multi-user access controls in electronic simultaneously editable document editor

Families Citing this family (37)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9230280B1 (en) 2013-03-15 2016-01-05 Palantir Technologies Inc. Clustering data based on indications of financial malfeasance
US8799799B1 (en) 2013-05-07 2014-08-05 Palantir Technologies Inc. Interactive geospatial map
US9785317B2 (en) 2013-09-24 2017-10-10 Palantir Technologies Inc. Presentation and analysis of user interaction data
US8812960B1 (en) 2013-10-07 2014-08-19 Palantir Technologies Inc. Cohort-based presentation of user interaction data
US9734217B2 (en) 2013-12-16 2017-08-15 Palantir Technologies Inc. Methods and systems for analyzing entity performance
US8832832B1 (en) 2014-01-03 2014-09-09 Palantir Technologies Inc. IP reputation
US9129219B1 (en) 2014-06-30 2015-09-08 Palantir Technologies, Inc. Crime risk forecasting
US9619557B2 (en) 2014-06-30 2017-04-11 Palantir Technologies, Inc. Systems and methods for key phrase characterization of documents
US9535974B1 (en) 2014-06-30 2017-01-03 Palantir Technologies Inc. Systems and methods for identifying key phrase clusters within documents
US9256664B2 (en) 2014-07-03 2016-02-09 Palantir Technologies Inc. System and method for news events detection and visualization
US9390086B2 (en) 2014-09-11 2016-07-12 Palantir Technologies Inc. Classification system with methodology for efficient verification
US9767172B2 (en) 2014-10-03 2017-09-19 Palantir Technologies Inc. Data aggregation and analysis system
US9229952B1 (en) 2014-11-05 2016-01-05 Palantir Technologies, Inc. History preserving data pipeline system and method
US9043894B1 (en) 2014-11-06 2015-05-26 Palantir Technologies Inc. Malicious software detection in a computing system
US9348920B1 (en) 2014-12-22 2016-05-24 Palantir Technologies Inc. Concept indexing among database of documents using machine learning techniques
US9335911B1 (en) 2014-12-29 2016-05-10 Palantir Technologies Inc. Interactive user interface for dynamic data analysis exploration and query processing
US9817563B1 (en) 2014-12-29 2017-11-14 Palantir Technologies Inc. System and method of generating data points from one or more data stores of data items for chart creation and manipulation
US9727560B2 (en) 2015-02-25 2017-08-08 Palantir Technologies Inc. Systems and methods for organizing and identifying documents via hierarchies and dimensions of tags
EP3070622A1 (en) 2015-03-16 2016-09-21 Palantir Technologies, Inc. Interactive user interfaces for location-based data analysis
US9886467B2 (en) 2015-03-19 2018-02-06 Plantir Technologies Inc. System and method for comparing and visualizing data entities and data entity series
US9996595B2 (en) 2015-08-03 2018-06-12 Palantir Technologies, Inc. Providing full data provenance visualization for versioned datasets
US9671776B1 (en) 2015-08-20 2017-06-06 Palantir Technologies Inc. Quantifying, tracking, and anticipating risk at a manufacturing facility, taking deviation type and staffing conditions into account
US9485265B1 (en) 2015-08-28 2016-11-01 Palantir Technologies Inc. Malicious activity detection system capable of efficiently processing data accessed from databases and generating alerts for display in interactive user interfaces
US9639580B1 (en) 2015-09-04 2017-05-02 Palantir Technologies, Inc. Computer-implemented systems and methods for data management and visualization
US9576015B1 (en) 2015-09-09 2017-02-21 Palantir Technologies, Inc. Domain-specific language for dataset transformations
US9424669B1 (en) 2015-10-21 2016-08-23 Palantir Technologies Inc. Generating graphical representations of event participation flow
US10114884B1 (en) 2015-12-16 2018-10-30 Palantir Technologies Inc. Systems and methods for attribute analysis of one or more databases
US9792020B1 (en) 2015-12-30 2017-10-17 Palantir Technologies Inc. Systems for collecting, aggregating, and storing data, generating interactive user interfaces for analyzing data, and generating alerts based upon collected data
US9652139B1 (en) 2016-04-06 2017-05-16 Palantir Technologies Inc. Graphical representation of an output
US10068199B1 (en) 2016-05-13 2018-09-04 Palantir Technologies Inc. System to catalogue tracking data
US10007674B2 (en) 2016-06-13 2018-06-26 Palantir Technologies Inc. Data revision control in large-scale data analytic systems
US9842338B1 (en) 2016-11-21 2017-12-12 Palantir Technologies Inc. System to identify vulnerable card readers
US10318630B1 (en) 2016-11-21 2019-06-11 Palantir Technologies Inc. Analysis of large bodies of textual data
US9886525B1 (en) 2016-12-16 2018-02-06 Palantir Technologies Inc. Data item aggregate probability analysis system
US10249033B1 (en) 2016-12-20 2019-04-02 Palantir Technologies Inc. User interface for managing defects
US10133621B1 (en) 2017-01-18 2018-11-20 Palantir Technologies Inc. Data analysis system to facilitate investigative process
US10133783B2 (en) 2017-04-11 2018-11-20 Palantir Technologies Inc. Systems and methods for constraint driven database searching

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
None *

Cited By (27)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10061828B2 (en) 2006-11-20 2018-08-28 Palantir Technologies, Inc. Cross-ontology multi-master replication
US9846731B2 (en) 2007-10-18 2017-12-19 Palantir Technologies, Inc. Resolving database entity information
US9501552B2 (en) 2007-10-18 2016-11-22 Palantir Technologies, Inc. Resolving database entity information
US9348499B2 (en) 2008-09-15 2016-05-24 Palantir Technologies, Inc. Sharing objects that rely on local resources with outside servers
US9880987B2 (en) 2011-08-25 2018-01-30 Palantir Technologies, Inc. System and method for parameterizing documents for automatic workflow generation
US9715518B2 (en) 2012-01-23 2017-07-25 Palantir Technologies, Inc. Cross-ACL multi-master replication
US9898335B1 (en) 2012-10-22 2018-02-20 Palantir Technologies Inc. System and method for batch evaluation programs
US10140664B2 (en) 2013-03-14 2018-11-27 Palantir Technologies Inc. Resolving similar entities from a transaction database
US10120857B2 (en) 2013-03-15 2018-11-06 Palantir Technologies Inc. Method and system for generating a parser and parsing complex data
US10152531B2 (en) 2013-03-15 2018-12-11 Palantir Technologies Inc. Computer-implemented systems and methods for comparing and associating objects
US9495353B2 (en) 2013-03-15 2016-11-15 Palantir Technologies Inc. Method and system for generating a parser and parsing complex data
US9852205B2 (en) 2013-03-15 2017-12-26 Palantir Technologies Inc. Time-sensitive cube
US9286373B2 (en) 2013-03-15 2016-03-15 Palantir Technologies Inc. Computer-implemented systems and methods for comparing and associating objects
US9996229B2 (en) 2013-10-03 2018-06-12 Palantir Technologies Inc. Systems and methods for analyzing performance of an entity
US10198515B1 (en) 2013-12-10 2019-02-05 Palantir Technologies Inc. System and method for aggregating data from a plurality of data sources
US10180977B2 (en) 2014-03-18 2019-01-15 Palantir Technologies Inc. Determining and extracting changed data from a data source
US10242072B2 (en) 2014-12-15 2019-03-26 Palantir Technologies Inc. System and method for associating related records to common entities across multiple lists
US9483546B2 (en) 2014-12-15 2016-11-01 Palantir Technologies Inc. System and method for associating related records to common entities across multiple lists
US10103953B1 (en) 2015-05-12 2018-10-16 Palantir Technologies Inc. Methods and systems for analyzing entity performance
US9661012B2 (en) 2015-07-23 2017-05-23 Palantir Technologies Inc. Systems and methods for identifying information related to payment card breaches
US9392008B1 (en) 2015-07-23 2016-07-12 Palantir Technologies Inc. Systems and methods for identifying information related to payment card breaches
US10127289B2 (en) 2015-08-19 2018-11-13 Palantir Technologies Inc. Systems and methods for automatic clustering and canonical designation of related data in various data structures
US9984428B2 (en) 2015-09-04 2018-05-29 Palantir Technologies Inc. Systems and methods for structuring data from unstructured electronic data files
US9760556B1 (en) 2015-12-11 2017-09-12 Palantir Technologies Inc. Systems and methods for annotating and linking electronic documents
US9514414B1 (en) 2015-12-11 2016-12-06 Palantir Technologies Inc. Systems and methods for identifying and categorizing electronic documents through machine learning
US10133588B1 (en) 2016-10-20 2018-11-20 Palantir Technologies Inc. Transforming instructions for collaborative updates
US10235533B1 (en) 2017-12-01 2019-03-19 Palantir Technologies Inc. Multi-user access controls in electronic simultaneously editable document editor

Also Published As

Publication number Publication date
GB201404489D0 (en) 2014-04-30
DE102014204834A1 (en) 2014-09-18

Similar Documents

Publication Publication Date Title
US9875293B2 (en) System and method for news events detection and visualization
Stvilia et al. A framework for information quality assessment
NL2012438B1 (en) Resolving similar entities from a database.
US5390359A (en) Storing and retrieving records in a computer system
US20120246154A1 (en) Aggregating search results based on associating data instances with knowledge base entities
US20150188872A1 (en) System and method for detecting confidential information emails
US8965914B2 (en) Grouping identity records to generate candidate lists to use in an entity and relationship resolution process
US9292388B2 (en) Determining and extracting changed data from a data source
DE102014204830A1 (en) Computer-implemented systems and methods for comparing and associating objects
DE102014204834A1 (en) Computer-implemented systems and methods for comparing and associating objects
US9785773B2 (en) Malware data item analysis
US8601059B2 (en) Sharing form training result utilizing a social network
US20120330959A1 (en) Method and Apparatus for Assessing a Person's Security Risk
US9619557B2 (en) Systems and methods for key phrase characterization of documents
WO2012135319A1 (en) Processing data in a mapreduce framework
US20160180557A1 (en) Systems and interactive user interfaces for dynamic retrieval, analysis, and triage of data items
CN102171702B (en) Detection of confidential information
EP2664997A2 (en) System and method for resolving named entity coreference
US8918394B2 (en) Generating candidate entities using over frequent keys
US8725494B2 (en) Signal processing approach to sentiment analysis for entities in documents
US9237161B2 (en) Malware detection and identification
WO2009087757A1 (en) Information filtering system, information filtering method, and information filtering program
JP5990284B2 (en) The system and method of spam detection using the Character histogram
US9195566B2 (en) Defect analysis system for error impact reduction
JP5600168B2 (en) A method and system for web page content filtering

Legal Events

Date Code Title Description
WAP Application withdrawn, taken to be withdrawn or refused ** after publication under section 16(1)