GB1590871A

GB1590871A - Data comparison apparatus

Info

Publication number: GB1590871A
Application number: GB2475278A
Authority: GB
Original assignee: Xerox Corp
Current assignee: Xerox Corp
Priority date: 1978-05-31
Filing date: 1978-05-31
Publication date: 1981-06-10

Description

(54) DATA COMPARISON APPARATUS (71) We, XEROX CORPORATION, of Xehox Square, Rochester, New York, United States of America, a corporation organised under the laws of the State of New York, United States of America, do hereby declare the invention, for which we pray that a patent may be granted to us, and the method by which it is to be performed, to be particularly described in and by the following statement: This invention relates to a data comparison apparatus and in particular to an apparatus for enabling records to be rapidly retrieved from a store of information by their contents.

In order to store large amounts of information efficiently and economically, several types of memory devices have been developed in recent years. Two such memory devices are the magnetic drum and the mag netic disc, which store information in the form of binary bits represented by the magnetisation state of material coating the surface of a moving member. Transducer heads are provided in close proximity to the moving surface to read or detect changes of magnetic flux as the surface passes by. The heads convert the changes of flux into electrical signals and also can be energised to writc, i.e., change the state of the magnetic flux, on the moving surface. The bits are stored on "tracks" on the drum or disc, a track being the area passing underneath a particular head during rotation of the drum or disc.

The reading operation does not destroy the magnetic state of the material passing under the heads, this state being changed only by a writing operation. Although such devices can store large amounts of information, the retrieval of information from the store has presented problems.

One way of accessing information stored in large capacity memories is one in which a record is accessed or identified by any portion of its contents. The memory device, when this accessing technique is used, is sometimes referred to as an associative memory device. Where the record contains human language, minor spelling errors or variations occur which would prevent a record being accessed in this way.

Thus it would be desirable to be able to access a record by an approximate match to a portion of its contents.

Accessing a record by any portion of its contents is most conveniently done by examining the whole of each record. To do this in a reasonable time with files of a practical size, it is necessary to perform the examination by the fastest means possible.

It is therefore an object of the present invention to provide a data comparison apparatus of the associative memory kind with tolerance to minor spelling errors and variations.

According to the present invention there is provided an apparatus for comparing a profile string of bit patterns with a source string of bit patterns to determine whether the profile string is matched to a predetermined extent with any succession of bit patterns.

forming part of the source string, comprising one or more shift registers arranged to circulate through each of their positions in succession the bit patterns of the profile string; a set of comparators, one for each position of the registers, arranged to compare in succession, and in synchronism with the circulation of the bit patterns of the profile string, each bit pattern of the source string with the instantaneous contents of all the positions of the registers, a set of counters responsive to the outputs of the comparators; and means responsive to the outputs of the counters to provide an indication of a match to said predetermined extent between the bit patterns of the profile string and of the source string.

By the use of the invention implemented in hardware and by the use of a storage device with a fast seri,al transfer speed, it is possible to search files of a practical size in a fraction of a second and at low cost.

An apparatus in accordance with the invention will now be described, by way of example, with reference to the accompanying drawings, in which: Figs. la to le are a set of diagrams of a simplified circuit for explaining the basic principles of the invention; Fig. 2 is a diagram illustrating a modification of the circuit of Fig. 1; and Fig. 3 is a more detailed circuit diagram of one embodiment of the invention.

Referring to Fig. 1, there is shown a set of five circuit diagrams Figs. la to le at successive time intervals. The circuit diagrams are in block form and are considerably simplified as an aid to the explanation of the principles of the invention.

It is assumed, by way of example, that it is desired to determine those records in which the word "START" appears in a store of information. The store may be a magneticdisc store, with the data arranged for very rapid serial read-out as a continuous string of characters (sub-divided into records). Each character is typically represented as a digital bit pattern which appears in parallel on a set of output lines. For alphanumeric characters, an eight bit pattern is often used. This source string of characters is then available for matching against a profile string of characters, in the present case the word "START". If and when the word (START) is found in the source string, the record or records containing the profile word are noted for subsequent use.

The profile string is shifted one character at a time in parallel through the positions 11, 12, 13, 14, 15 . . . of the parallel set of shift registers represented in the drawings as a single shift register 10. The source string of characters is supplied to a source bus 20, whereby each character of the source string appears for an appropriately clocked period of time on one input of every one of a set of comparators 21, 22, 23, 24, 25 The clocking periods of the characters of the source string are synchronised with the shifting of the characters of the profile string through the shift register 10. The character contained in position 11 of the shift register 10 is compared with the character of the source string in comparator 21, and similar comparisons are made for position 12 by comparator 22, for position 13 by comparator 23, and so on. In the present example, it is assumed that the letters of the word "START" appear as shown in Fig. la positions 11, 12, 13, 14 and 15 respectively of the shift register 10 at the same time as a letter 'S' appears at the beginning of the word "START" in the source string. In these circumstances, both inputs of comparator 21 are an 'S', so the comparator provides an output indicated as a "1", which is duly counted by a counter 31. Similar counters 32, 33, 34, and 35 are associated with comparators 22, 23, 24 and 25 respectively. In the second comparator 22, the letter 'S' is compared with 'T', so there is no output from that comparator. Thus, counter 32 remains at a zero count. Similarly, comparators, 23, 24 and 25 are all comparing an 'S' with, respectively, an 'A', an 'R' and a 'T'. Thus, counters 33, 34 and 35 remain at zero count. The net result of the comparisons carried out in the first time period indicated by Figure la is a count of 1 on counter 31, and zero counts on all the other counters.

Figure 1b indicates the position during the next clocking period, after all the characters in the profile string have been shifted by one in an upwardly direction.

Thus, positions 11, 12, 13 and 14 now contain the letters T A R and T respectively.

Position 15 is assumed to be empty. During the same clocking period, the next character of the source string, T, has been selected so that it now appears on the source bus 20. In these circumstances, it will be seen that comparator 21 has an output, thereby advancing counter 31 to a count of 2. Comparators 22 and 23 are comparing a letter T with letters A and R respectively, and so have no output. Accordingly, counters 32 and 33 remain at zero count. Comparator 24, however, is comparing a T with a T, and so has an output, so that counter 34 is advanced to a count of 1. There is no match, of course, in comparator 25, so counter 35 remains at zero count. Thus the net result in the second clocking period is that the count in counter 31 advances to 2, and the count in counter 34 advances to 1.

Figure le indicates the position during the third clocking period. In this period, the profile string has shifted such that letters A, R and T appear in positions 11, 12 and 13 of the shift register 10. Letter A now appears on the source bus 20, so a match is achieved in comparator 21, advancing the count on counter 31 to 3. No further matches are achieved, so the other counters remain as they were after the second clocking period.

Fig. 1d indicates the position in the fourth clocking period, when letters R and T appear in positions 11 and 12 of the shift register 10, and letter R appears on the source bus 20. In these circumstances, comparator 21 adds another count to counter 31, making a total of 4. None of the other counters are changed, since no further matches occur. Once again, it is assumed for simplicity that no further characters are following the word "START" in the profile string.

Figure le indicates the position in the fifth and final clocking period. The final T of the profile word now appears in position 11 of the shift register 10. This coincides with the appearance of the final letter T of the word "START" in the source string, so a further count is added by comparator 21 to counter 31 making a total count of 5 on counter 31.

Thus, the final position is that counter 31 has a count of 5, and counter 34 has a count of 1. The other counters remain at zero throughout. The fact that any one of the counters has counted to 5 indicates, for a five-letter profile word that an exact match has been made between the profile word and a word in the source string. If the matching words of the profile and source strings had coincided in a different way, the only effect would have been that the count of 5 would have appeared on a different counter. It is thus only necessary to set the threshold of the counters to the same number as the number of letters in the profile word to achieve exact matching between profile words and source words.

It is possible to build in a degree of tolerance to errors by setting the thresholds of the counters to be less than the number of letters in the profile word. Thus, in the example just described, it would be possible to set the counter thresholds to 4 such that an indication is provided when appropriate four-letter matches are achieved. This indication would be provided in the present example if the word "STAR" or the word "TART" appeared on the source string.

Similarly, if the counter thresholds were set to only three, then three-letter matches would be possible, with an indication being given if words such as "TAR" or "ART" appeared on the source string.

As can be seen, in the simple example discussed above, if the profile word is shorter than a word containing the same characters in the source string (e.g. STARTLE) then an exact match will occur in any event between the profile word and the first five letters of the word in the source string. (This is chosen by way of example only; in practice, the space at the end of the profile word will normally constitute a character. However, if in this case, the counter thresholds are set to 4, then a match will be indicated.) If a substitution occurs in either the profile word or the source word, it is necessary to take advantage of the error-tolerating capability if a match is to be indicated. Thus, for example, if the profile word is START and the word STARK appears on the source string, it is possible to obtain a match indication if the counter thresholds are set to 4. Such a match may be said to be an inexact match.

A simple extension of the error-tolerating capability just described may be provided to cope with omissions of characters, insertions, or transpositions. The effect of, for example, an omission can be seen by considering what happens when the misspelled word STRT in the store is compared with a profile word START. Referring back to Figure 1 the first two comparisons (diagrams A and B) are the same as described above. Thus, counter 31 at this point has a count of 2, and counter 34 has a count of 1. The next step is the arrival of letters A, R and T in register positions 11, 12 and 13 and the arrival of an R on the source bus 20. A match therefore occurs in comparator 22 and a count of one is recorded by counter 32. In the next time period, a T appears on the source bus 20 and letters R and T are in positions 11 and 12 of the register 10. Thus another match occurs in comparator 22, so counter 32 advances to 2.

Thus, in the situation juhst described, counts of two occur in each of adjacent counters 31 and 32. In order to enable the apparatus to provide a match indication in such circumstances, a set of OR gates 41, 42, 43 . - . are employed, as shown in Fig.

2, each having inputs from two adjacent comparators, and each having an output to one counter. If the counter thresholds are set at 4 (for a five letter word) a match indication (inexact) will be provided whenever four matches are recorded by a single comparator, and also whenever a total of four matches are recorded by any two adjacent comparators. Such an arrangement therefore allows a match indication to be provided when a letter is missing from the source word. Similarly, if there is an addi tional character in either the source word or the profile word a match indication can be provided if the OR gates are employed.

Furthermore, it can be seen that transpositions of two letters in either the profile word or the source word will still allow a match indication to be provided when tthe OR gates are used.

Although in the above description 5-letter words have been chosen (for simplicity) it is to be understood that in practice words of any length or groups of words may be used.

A useful provision for profile word length is twenty characters, although it is not necessary to fill all the spaces with characters in such a case. As mentioned above, a space can count as a character for matching purposes, as can punctuation and other marks.

Referring now Fig. 3, there is shown a more complete circuit diagram of a device for exact and partial matching in record retrieval.

The device is operated by a small programmable computer known as a system control unit (SCU) (not shown), the input/ output bus 50 of which is suitably connected to the device by way of an interface unit 51. The operator enters the profile word to be matched via a keyboard connected to the SCU. This profile word is passed via a profile and threshold register 52, under the control of a load/circulate control 53, to the profile register 54. The profile register 54 consists of a string of shift registers which may typically consist of four 40-position registers. If each character is represented by an eight bit pattern, then each of the four sections of the register will hold five characters, as shown. Thus, in total, the profile register 54 has twenty positions. The profile word is circulated by control 53, synchronisation with source string characters being provided by tiring pulses from the disc organiser (not shown) by way of line 55.

Searchable record data (the source string) is passed as a continuous stream of characters from the disc memory to the matching logic by way of source bus 56. A set of eight-bit comparators 57 are arranged to compare, in each clocking period, the character appearing on the source string with the character appearing in each position of the profile register. Individual character matches are recorded by hit counters 58, one of which is associated with each comparator.

When the comparisons have been made on a source character, the characters in the profile register are shifted one place (upwardly in Fig. 3). The next character on the source string then becomes available and is compared. When a word match occurs, the comparator and counter operating in the position of the leading profile character will record all the character matches for that word. The effect of the profile shift is thus to synchronise matching characters. Character matches not in the correct sequence will put up counts on other counters. These counts will subsequently be ignored. An exact word match will register a count on one of the hit counters equal to the profile length. When the full count is reached a hit insult is indicated to the SCU by raising a "match result ready" flag line 59. A suitable indication is provided to the SCU by way of match result flag latch 60.

When inexact matching is required, the operator may enter the match threshold. The match threshold may be a decimal number representing the percentage of characters expected to match. The threshold is set on a suitable switch (not shown) the setting of which is communicated by way of line 61 to a threshold inpul register 62. The setting is entered by the SCU into the profile and threshold register 52, the threshold being set on counters 58 by way of line 63. The SCU will compute the required count and provide the necessary start count for the hit counters. Thus, the counters will be preset to a count of m-n, where m is the counter modulo and n is the required number of character matches for the given profile word. Each hit counter is pre-set individually by reset registers 64. The reset registers are a string of shift registers which circulate a reset bit in step with the profile register. The pre-set occurs in the last profile character position. Timing is so organised that hits from the last character will be registered prior to the reset period.

The hit counters 58 are bi-directional.

Mode select logic is provided, by means of count mode control 65 which enables three modes of hit counting. The count mode is selected by the operator, and is entered from the SCU via a count mode register 66.

The hit counters will count up for a character match in all modes. For mis-matching characters, however, the counters may be set to operate in one of three modes. In the first mode, mis-matching characters cause the counter to count down. In the second mode, the counter holds its existing count for mismatching characters. In the third mode, a mis-matching character causes the counter to be pre-set again.

The selectivity of the match achieved is controlled by the count mode, the counter threshold, and the profile entry. For further control over the matching, it is possible to incorporate a set of OR gates as described above with reference to Fig. 2.

In a practical system, there is no need for the device to distinguish between upper and lower case characters for matching purposes.

Lower case source string characters may accordingly be replaced by upper case characters by stripping logic (not shown).

Profile words are entered in upper case only, although it may be arranged for a visual display unit to use the full character set.

Unmatchable characters may be introduced into both character strings for padding purposes. -For example a NULL may be used as a padding character in the profile string, and a DEL in the source string. In the embodiment described with reference to Fig. 3, the profile word may be from one to twenty characters long. For words of less than twenty matchable characters, padding characters are used to bring the number of characters up to twenty. These padding characters will normally be added before the first matchable character in the profile string. The source string will also contain matchable and padding characters. The stripping logic will recognise binary fields, formatting characters and delimiters. It will remove any such characters, replacing them with padding characters before passing the data to the matching logic via source bus 56.

There are many fields of application of the apparatus of the invention. For example, the apparatus is useful in any cataloguing.

system, since information can be stored in an unstructured fashion and rapidly retrieved by examination of the whole contents of the store. The inexact matching capability of the apparatus is very desirable in such an application in that minor differences between the store and a profile word will not prevent a match being made.

Thus, errors made in constructing the store can be tolerated, as can errors made in setting up the profile word. Furthermore, apart from the ability to deal with errors, the apparatus can also cope with legitimate spelling variations, differences between singular and plural words, and differences in prefixes and suffixes. Such variations would prevent the retrieval of the desired information if exact matching techniques were used.

One such use of the apparatus of the invention in cataloguing is in a library. All information relating to a given book, periodical, or article, can be entered into the store, including such details as date of ordering, name of supplier, cost, order number, bibliographic details, date of receipt in the library, classification or location details, dates of loans, and names of persons wanting the book when it is next available. Any of these items of information can be retrieved by a search on the whole contents of the store, the inexact matching capabilities of the invention being particularly useful in the light of the traditional vagueness of library users on such information as book titles, authors and publishers.

Another kind of application of the apparatus is in, for example, voice pattern recognition, optical character recognition, human tissue type matching, or finger print matching. Electronic analyses of the relevant pattern provide the bit patterns which can be used in a matching operation, the inexact matching capability of the apparatus providing a very powerful aid.

In connection with optical character recognition, the invention may advantageously be applied in the field of identifying Japanese or similar writing. It is particularly difficult to design automatic office information systems for the Japanese (and similar) markets in that typewritten material does not normally form the basis of office communication. This results from the ideographic nature of the Japanese language which, unlike the Romance languages, may demand a 'keyboard' offering a thousand or more characters. This makes typing a tedious process and consequently all but the most formal communications are oral or handwritten.

In order to take advantage of electronic text editing, storage and communication functions offered by present day systems, some means is required for capturing Kanji characters in electronic form suitable for display on a screen. In Romance languages this is done with a normal typewriter key board. A Japanese user, however, would prefer to write Kanji. This is much quicker and, indeed, more satisfying in the context of Japanese calligraphic culture.

Thus, instead of the normal keyboard, there is provided a writing surface on which the user forms Kanji characters. This can be done with a light pen and soft display screen or, perhaps more satisfyingly, a paper, brush and ink system scanned through a support pattern.

The character, however written, needs to be identified and presented in a stylised form on an editing screen. This is done with the aid of the inexact matching capabilities of the apparatus of the invention. The essen tial features of the written character, perhaps reduced to "spine functions" which an define the elements of an ideogram, are compared with a stored dictionary of such functions. Exact matching is not re quired in the comparison, and the process will select and display the most closely matching ideogram. The user can then quickly delete all but the one that he needs.

The comparison process is fast (of the order of a few hundred milliseconds) and the ex tent of matching is pre-selectable from exact matching to, say, fifty percent correspon dence. The whole process of entering a character on the screen takes only a few seconds, and the hardware is less cumber some then, and just as rapid as, a system which incorporates some form of Japanese typewriting device.

WHAT WE CLAIM IS:- 1. Apparatus for comparing a profile string of bit patterns with a source string of bit patterns to determine whether the profile string is matched to a predetermined extent with any succession of bit patterns forming part of the source string, compris ing one or more shift registers arranged to circulate through each of their positions in succession the bit pattenls of the profile string; a set of comparators, one for each position of the registers, arranged to com pare in succession, and in synchronism with the circulation of the bit patterns of the pro file string, each bit pattern of the source string with the instantaneous contents of all the positions of the registers, a set of counters responsive to the outputs of the comparators; and means responsive to the outputs of the counters to provide an in dication of a match to said predetermined extent between the bit patterns of the pro file string and of the source string.

2. Apparatus according to Claim 1 wherein the counters are arranged to be incremented for each match determined by their associated comparators, the counters having a threshold which is setable as a pro

**WARNING** end of DESC field may overlap start of CLMS **.

Claims

**WARNING** start of CLMS field may overlap end of DESC **. an unstructured fashion and rapidly retrieved by examination of the whole contents of the store. The inexact matching capability of the apparatus is very desirable in such an application in that minor differences between the store and a profile word will not prevent a match being made. Thus, errors made in constructing the store can be tolerated, as can errors made in setting up the profile word. Furthermore, apart from the ability to deal with errors, the apparatus can also cope with legitimate spelling variations, differences between singular and plural words, and differences in prefixes and suffixes. Such variations would prevent the retrieval of the desired information if exact matching techniques were used. One such use of the apparatus of the invention in cataloguing is in a library. All information relating to a given book, periodical, or article, can be entered into the store, including such details as date of ordering, name of supplier, cost, order number, bibliographic details, date of receipt in the library, classification or location details, dates of loans, and names of persons wanting the book when it is next available. Any of these items of information can be retrieved by a search on the whole contents of the store, the inexact matching capabilities of the invention being particularly useful in the light of the traditional vagueness of library users on such information as book titles, authors and publishers. Another kind of application of the apparatus is in, for example, voice pattern recognition, optical character recognition, human tissue type matching, or finger print matching. Electronic analyses of the relevant pattern provide the bit patterns which can be used in a matching operation, the inexact matching capability of the apparatus providing a very powerful aid. In connection with optical character recognition, the invention may advantageously be applied in the field of identifying Japanese or similar writing. It is particularly difficult to design automatic office information systems for the Japanese (and similar) markets in that typewritten material does not normally form the basis of office communication. This results from the ideographic nature of the Japanese language which, unlike the Romance languages, may demand a 'keyboard' offering a thousand or more characters. This makes typing a tedious process and consequently all but the most formal communications are oral or handwritten. In order to take advantage of electronic text editing, storage and communication functions offered by present day systems, some means is required for capturing Kanji characters in electronic form suitable for display on a screen. In Romance languages this is done with a normal typewriter key board. A Japanese user, however, would prefer to write Kanji. This is much quicker and, indeed, more satisfying in the context of Japanese calligraphic culture. Thus, instead of the normal keyboard, there is provided a writing surface on which the user forms Kanji characters. This can be done with a light pen and soft display screen or, perhaps more satisfyingly, a paper, brush and ink system scanned through a support pattern. The character, however written, needs to be identified and presented in a stylised form on an editing screen. This is done with the aid of the inexact matching capabilities of the apparatus of the invention. The essen tial features of the written character, perhaps reduced to "spine functions" which an define the elements of an ideogram, are compared with a stored dictionary of such functions. Exact matching is not re quired in the comparison, and the process will select and display the most closely matching ideogram. The user can then quickly delete all but the one that he needs. The comparison process is fast (of the order of a few hundred milliseconds) and the ex tent of matching is pre-selectable from exact matching to, say, fifty percent correspon dence. The whole process of entering a character on the screen takes only a few seconds, and the hardware is less cumber some then, and just as rapid as, a system which incorporates some form of Japanese typewriting device. WHAT WE CLAIM IS:-

1. Apparatus for comparing a profile string of bit patterns with a source string of bit patterns to determine whether the profile string is matched to a predetermined extent with any succession of bit patterns forming part of the source string, compris ing one or more shift registers arranged to circulate through each of their positions in succession the bit pattenls of the profile string; a set of comparators, one for each position of the registers, arranged to com pare in succession, and in synchronism with the circulation of the bit patterns of the pro file string, each bit pattern of the source string with the instantaneous contents of all the positions of the registers, a set of counters responsive to the outputs of the comparators; and means responsive to the outputs of the counters to provide an in dication of a match to said predetermined extent between the bit patterns of the pro file string and of the source string.

portion of the number of bit patterns in the profile string, and being arranged to provide an output when the threshold is reached.

3. Apparatus according to Claim 2, wherein the counters are arranged to count down for mis-matching bit patterns.

4. Apparatus acording to Claim 2, wherein the counters are arranged to hold their existing counts for mis-matching bit patterns.

5. Apparatus according to Claim 2, wherein the counters are arranged to re-set to a predetermined count for mis-matching bit patterns

6. Apparatus according to any one of Claims 1 to 5 including a set of OR gates having outputs connected respectively to the counters, and each having inputs from two adjacent comparators.

7. Apparatus for comparing a profile string of bit patterns with a source string of bit patterns, substantially as hereinbefore described with reference to the accompanying drawings.

8. Apparatus for identifying handwritten characters comprising a writing surface, means for reducing a character written on said surface to a profile string of bit patterns, and means for comparing this profile string of bit patterns with a source string of bit patterns derived from a dictionary store, the comparing means comprising the apparatus of any one of Claims 1 to 7.