WO2004081819A1 - A method and system for pattern matching - Google Patents

A method and system for pattern matching Download PDF

Info

Publication number
WO2004081819A1
WO2004081819A1 PCT/IN2004/000059 IN2004000059W WO2004081819A1 WO 2004081819 A1 WO2004081819 A1 WO 2004081819A1 IN 2004000059 W IN2004000059 W IN 2004000059W WO 2004081819 A1 WO2004081819 A1 WO 2004081819A1
Authority
WO
WIPO (PCT)
Prior art keywords
pattern
source
sequence
target
group leader
Prior art date
Application number
PCT/IN2004/000059
Other languages
French (fr)
Inventor
Sriram K. N. V. Kumar
Rama Shankar Mantha
Chandrasekhar Sarasvat Revur
Original Assignee
Hewlett-Packard Development Company, L.P.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hewlett-Packard Development Company, L.P. filed Critical Hewlett-Packard Development Company, L.P.
Priority to EP04720131A priority Critical patent/EP1602041A1/en
Priority to JP2006507626A priority patent/JP2006522401A/en
Publication of WO2004081819A1 publication Critical patent/WO2004081819A1/en

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F7/00Methods or arrangements for processing data by operating upon the order or content of the data handled
    • G06F7/02Comparing digital values
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2207/00Indexing scheme relating to methods or arrangements for processing data by operating upon the order or content of the data handled
    • G06F2207/02Indexing scheme relating to groups G06F7/02 - G06F7/026
    • G06F2207/025String search, i.e. pattern matching, e.g. find identical word or best match in a string

Definitions

  • Pattern matching is the process of finding some or all of the occurrences of a target pattern in a source pattern.
  • Compressed pattern matching is the process of finding some or all of the occurrences of a target pattern in a compressed source pattern without decompressing the source pattern.
  • Pattern matching is the application of analytical rules to a block of data to identify a feature of that block of data.
  • the most common pattern matching problem is the process of finding some or all occurrences of a sequence of elements [Y1...Ym] (target pattern) within a larger sequence of elements [X1...Xn] (source pattern).
  • the elements come from a finite element set - an alphabet set.
  • the set may be the English alphabet, ⁇ 0,1 ⁇ , natural numbers, etc.
  • the most popular algorithms for this problem are the Knuth-Morris-Pratt algorithm, the Boyer-Moore algorithm and the Rabin-Karp algorithm.
  • Pattern matching is used for simple text search, searching for data in image data, speech data, video data, audio data, bio-medical sequence analysis, etc.
  • Data compression is mainly used for reducing storage space and to speed up data transmission.
  • Various forms of compression are known. Of particular interest is arithmetic coding compression for which compressed pattern matching has not heretofore been possible. Arithmetic coding originated in the 1970s and 1980s (see for example US 4,122,440). Arithmetic coding is used in several applications, including Speech and Medical Image compression.
  • compressed pattern matching the pattern matching is performed in the compressed domain.
  • compressed pattern matching of, for example, text strings can be stated as:
  • ac be a given compression algorithm
  • ac(D) be the result of ac compressing data D Input: compressed text ac(T) and compressed pattern ac(P)
  • Output all or some locations in T where pattern P occurs.
  • a method of determining whether a target pattern is present within a source pattern composed of one or more characters from an alphabet set by determining whether the position of the source pattern within a sequence of possible patterns is a position which correlates with a position within the sequence of possible patterns that includes the target pattern.
  • a data processing apparatus to determine whether a target pattern is present within a source pattern composed of one or more characters of an alphabet, comprising: i. a first memory for storing a target pattern; ii. a second memory for storing a source pattern; iii. a processing means for determining the position of the source pattern in a sequence of possible patterns; iv. a processing means for determining the position of the target pattern in the sequence of possible patterns; and v. a processing means for correlating the source position with the target position.
  • a data processing apparatus to determine whether a target sequence is present within a source sequence composed of one or more characters of an alphabet, comprising: i. a first memory for storing a target sequence position wherein the target sequence position is the position of the target sequence within a lexicon of all possible combinations of characters of the alphabet; ii. a second memory for storing a source sequence position wherein the source sequence position is the position of the source sequence within the lexicon; iii.
  • a processing means for computing a set of positions of sub-sequences of the source sequence wherein the position of the sub-sequence is the position of the sub-sequence within the lexicon and wherein the subsequence includes the first character position of the source sequence; iv. a processing means for determining a series defining all positions of sequences within the lexicon which contain the target sequence; v. a processing means for correlating the set with the series.
  • Figure 1 is a chart of positions of source pattern strings of length 4, containing the target pattern '1', where the alphabet set is ⁇ 0,1 ⁇ ;
  • Figure 2 is a chart of source pattern strings of length 4, containing the target pattern 'b', where the alphabet set is ⁇ a,b,c,d,e ⁇ ;
  • Figure 3 is a chart of position numbers of source pattern strings of length 4, containing the target pattern 'fo', where the alphabet set is ⁇ a,b,c,d,e ⁇ ;
  • Figure 4 is a chart as in figure 3, where the position numbers are set out to scale.
  • the source pattern string could contain zero, one or several matches of the target pattern string.
  • Figure 1 shows a simple scenario for the purpose of illustration.
  • the alphabet set ⁇ 0,1 ⁇ and a source data length, L t , of 4.
  • the possible source pattern strings are then all binary numbers from 0000 to 1111.
  • the positions of the source pattern strings in a numerically ordered set are also shown in the figure.
  • the position numbers of source pattern strings satisfying the pattern match conditions are shown for the four possible target pattern positions within the source pattern string. For example, in row 4 the target pattern '1' occupies the first position in the source pattern string. This condition is satisfied for the binary numbers 1000 to 1111 , which have position numbers from 9 to 16.
  • the position numbers of source pattern strings where the target pattern occurs form one or several groups of successive numbers with breaks between groups.
  • the number of elements in each group depends on the row, and hence the location of the target pattern within the source pattern string.
  • each group contains a single element, with a gap of 1 between each group.
  • each group contains 4 elements, with a gap of 4 between each group.
  • the elements of row 1 form an arithmetic series, where the starting element, b 0 , is equal to the position of the target pattern, P p , and the difference is the number of elements in the alphabet set, N, raised to the power of the length of the target pattern (i.e. the number of elements in the target pattern), L p :
  • each row then has a group leader series.
  • the group leader series of any row is related to the group leader series of an adjacent row by a factor of N, the number of elements in the alphabet set.
  • the algorithm searches for pattern matches at each possible target pattern position within the source pattern string, that is one row at a time.
  • L iter (2) which is incremented or decremented by 1 to move one row at a time.
  • this algorithm searches for all pattern matches in the source data. However, a similar algorithm could easily search for a single pattern match, ending immediately after a pattern match is found. In this case, empirical knowledge of the strings involved could significantly speed up the algorithm.
  • the sequence of searching may depend upon a characteristic of the source pattern string. If it were known a priori that the target pattern was likely to be contained towards the end of the source pattern string we should start with the assumption that the target pattern was contained in row 1 and move progressively one row at a time towards row (L D + 1 ). On the other hand, if the target pattern was likely to be contained towards the beginning of the source pattern string, we would move progressively from row (L D + 1) to row 1.
  • Another similar algorithm could search for a pattern match at a particular position in the source pattern string.
  • the sequence of searching may be based upon an analysis of the source pattern string and occur according to the estimated probability for each row. It will be appreciated that a range of search sequences may be employed depending upon characteristics of the source data.
  • the step of calculating the group leader position can be achieved as follows.
  • the position, P j of the lowest member of each possible group can be represented by:
  • N is the number of elements in the alphabet and r is the row number.
  • P p is the position of the target pattern within an ordered sequence of possible target patterns and L p is the number of elements in the target pattern.
  • N Number of elements in the alphabet. 7 - A source pattern string. P - A target pattern string. L t - Length of T.
  • R h High value of the range of ac output for a particular string.
  • a string is represented by an interval on a number line.
  • the size of the interval is determined by the probabilities of the symbols of the alphabet. In the equiprobable case, it is a simple matter to calculate the position number of a string:
  • R h is the highest value of the interval and R ⁇ is the lowest value of the interval.
  • the position number can still be determined, although this is not as straightforward as in the equiprobable case.
  • the source sequence can be represented as a position in a list of all possible combinations of the elements of the alphabet with the length of the source sequence - the source lexicon.
  • the source sequence itself is in fact that position when represented in base n (n is the number of elements in the alphabet) where the elements of the alphabet represent digits in base n.
  • a set of the positions of all sub-sequences of the source sequence which include the first character position of the source sequence, within a lexicon of all possible combinations of the elements of the alphabet with the length of the sub-sequence, can be computed.
  • the set will contain the following elements: the first digit of the source sequence in base n, the first digit and the second digit, the first digit and the second and the third, and so on for the length of the source sequence position in base n.
  • "words" containing the target sequence are n to the power of the length of the target sequence apart. Using the position of the first "word” containing the target sequence, a series of positions of "words" containing the target sequence can therefore be defined.
  • the position of the element in the set which also matches with the target t series describee the position in the source sequence of the rightmost element of the target sequence.
  • ⁇ et of source sub-sequence positions may be matched against the series of possible target positions in O(n) by using the remainder equation.
  • a target sequence tX....Xm] is present within a e ⁇ ure ⁇ s ⁇ qu ⁇ r ⁇ [Y ⁇ ...YJ u ⁇ r ⁇ ⁇ X ⁇ ..._ an
  • the method can be implemented using a range of standard data processing devices or specialised graphics audio processing or sequencing devices.
  • the method may be deployed on a personal computer operating under the MICROSOFT WINDOWSTM environment or on a server machine operating under the UNIX operating system.
  • the method may be implemented in software executing on a device or implemented in specialised hardware.
  • the present invention represents the first method for pattern matching of arithmetically compressed source data without decompression of the data.
  • the invention can also operate using a compressed target pattern string.
  • the invention should be of immense utility at arithmetic decoder stations, where the decoder could typically check for partial or full matches with any prior strings of data.
  • the invention can be used to find all pattern matches, one pattern match or a pattern match at a particular position in the source pattern string.
  • the invention can also be executed in parallel, as a search on a particular interval can be performed independently of searches on other intervals. That is, the invention can be applied to more than one source string at the same time.
  • the invention could also operate in a decentralised system, for example a search engine accessible over a communications network.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computational Mathematics (AREA)
  • Mathematical Analysis (AREA)
  • Mathematical Optimization (AREA)
  • Pure & Applied Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Compression, Expansion, Code Conversion, And Decoders (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

Pattern matching can be achieved by considering only the position numbers of a source pattern and a target pattern within ordered sequences of possible source patterns and target patterns respectively. The position numbers of source patterns containing the target pattern form a number of groups. The number of source patterns within each i group and the number of source patterns in the gaps between groups depend on the position of the target pattern within the source pattern, the length of the target pattern and the number of elements in the alphabet set. Each group also has a position number, its position within an ordered sequence of groups. The group position number of an input source pattern is compared to a series derived from the position number of the target pattern, the length of the target pattern and the number of elements in the alphabet set (9). If the group position number is a member of the series (10), then the source pattern contains the target pattern (11). Pattern matching can be performed on uncoded data or Oh arithmetic coded data.

Description

A METHOD AND SYSTEM FOR PATTERN MATCHING
FIELD OF THE INVENTION
The invention relates to pattern matching. Pattern matching is the process of finding some or all of the occurrences of a target pattern in a source pattern. Compressed pattern matching is the process of finding some or all of the occurrences of a target pattern in a compressed source pattern without decompressing the source pattern.
BACKGROUND OF THE INVENTION
Pattern matching is the application of analytical rules to a block of data to identify a feature of that block of data.
The most common pattern matching problem is the process of finding some or all occurrences of a sequence of elements [Y1...Ym] (target pattern) within a larger sequence of elements [X1...Xn] (source pattern). The elements come from a finite element set - an alphabet set. The set may be the English alphabet, {0,1}, natural numbers, etc. The most popular algorithms for this problem are the Knuth-Morris-Pratt algorithm, the Boyer-Moore algorithm and the Rabin-Karp algorithm.
There are a number of pattern matching tools in the public domain. The most prominent of these are the grep family.
Pattern matching is used for simple text search, searching for data in image data, speech data, video data, audio data, bio-medical sequence analysis, etc.
Data compression is mainly used for reducing storage space and to speed up data transmission. Various forms of compression are known. Of particular interest is arithmetic coding compression for which compressed pattern matching has not heretofore been possible. Arithmetic coding originated in the 1970s and 1980s (see for example US 4,122,440). Arithmetic coding is used in several applications, including Speech and Medical Image compression.
In compressed pattern matching, the pattern matching is performed in the compressed domain. Succinctly put, compressed pattern matching of, for example, text strings can be stated as:
Let ac be a given compression algorithm, let ac(D) be the result of ac compressing data D Input: compressed text ac(T) and compressed pattern ac(P)
Output: all or some locations in T where pattern P occurs.
Amir and Benson (" Efficient Two-dimensional Compressed Matching", in Proc. Second IEEE Data Compression Conference, pp 279 - 288, March 1992) showed that complexity of pattern matching is reduced in the compressed domain compared to any conventional method in the uncompressed domain.
However, it has not been considered possible to perform pattern matching on "arithmetic codes" without decompressing that data ("TR-COSC 07/01 Pattern Matching in Compressed Texts and Images", by Tim Bell et al, Technical Reports, University of Canterbury, available at http://www.cosc.cantarbu ^.ac.n2-/research/reøorJ:s TechReps/2001/tr Q1Q7.pdf
SUMMARY OF THE INVENTION
It is an object of the present invention to provide a method of pattern matching suitable for matching arithmetic codes or to at least provide the public with a useful choice.
According to a first aspect of the invention there is provided a method of determining whether a target pattern is present within a source pattern composed of one or more characters from an alphabet set by determining whether the position of the source pattern within a sequence of possible patterns is a position which correlates with a position within the sequence of possible patterns that includes the target pattern. According to a further aspect of the invention there is provided a method of determining whether a target sequence [X1...Xm] is present within a source sequence [Y1...Yq] where {X1...Xm} and {Y1...Yq} are members of a finite ordered set {Z1...Zo}, including the steps of: i. constructing a set of source sub-sequence values in base o
{[Vι], ... [V....VJ, ... [V, ...Vn]} where Vk = j : Yk = Zj; and ii. determining whether any values of the set correspond to a series of target values {p,...p+kom,...p+nom} where p is the number [D1...Dn] in base o such that Dk = h : X = Zh.
According to another aspect of the invention there is provided a data processing apparatus to determine whether a target pattern is present within a source pattern composed of one or more characters of an alphabet, comprising: i. a first memory for storing a target pattern; ii. a second memory for storing a source pattern; iii. a processing means for determining the position of the source pattern in a sequence of possible patterns; iv. a processing means for determining the position of the target pattern in the sequence of possible patterns; and v. a processing means for correlating the source position with the target position.
According to a further aspect of the invention there is provided a data processing apparatus to determine whether a target sequence is present within a source sequence composed of one or more characters of an alphabet, comprising: i. a first memory for storing a target sequence position wherein the target sequence position is the position of the target sequence within a lexicon of all possible combinations of characters of the alphabet; ii. a second memory for storing a source sequence position wherein the source sequence position is the position of the source sequence within the lexicon; iii. a processing means for computing a set of positions of sub-sequences of the source sequence wherein the position of the sub-sequence is the position of the sub-sequence within the lexicon and wherein the subsequence includes the first character position of the source sequence; iv. a processing means for determining a series defining all positions of sequences within the lexicon which contain the target sequence; v. a processing means for correlating the set with the series.
BRIEF DESCRIPTION OF THE DRAWINGS
The invention will now be described by way of example with reference to the accompanying drawings, in which:
Figure 1 is a chart of positions of source pattern strings of length 4, containing the target pattern '1', where the alphabet set is {0,1};
Figure 2 is a chart of source pattern strings of length 4, containing the target pattern 'b', where the alphabet set is {a,b,c,d,e};
Figure 3 is a chart of position numbers of source pattern strings of length 4, containing the target pattern 'fo', where the alphabet set is {a,b,c,d,e};
Figure 4 is a chart as in figure 3, where the position numbers are set out to scale; and
Figur© S is a flowchart showing the pattern matching algorithm.
DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION
The invention will be described in relation to examples where data could be either compressed or uncompressed.
Consider a source pattern string, t, and a target pattern string, p. These strings have lengths, Lt and Lp respectively, and we require that Lt≥ Lp (note that the invention can be applied to strings of arbitrary length, provided this condition is fulfilled). We define the position of the source pattern string, Ptl within an ordered series of source pattern strings. For example, for Lt = 4, and with an alphabet set {a,b,c,d,e}, the first 11 source pattern strings and their positions in a lexically ordered set are shown in table 1.
Figure imgf000007_0001
Table 1
Similarly, we define the position of the target pattern string, Pp. For Lp = 1 , and with the same alphabet set {a,b,c,d,e}, the possible target pattern strings and their positions in a lexically ordered set are shown in table 2.
Figure imgf000007_0002
Table 2
The source pattern string could contain zero, one or several matches of the target pattern string.
Figure 1 shows a simple scenario for the purpose of illustration. In figure 1 we begin with the alphabet set {0,1} and a source data length, Lt, of 4. The possible source pattern strings are then all binary numbers from 0000 to 1111. The positions of the source pattern strings in a numerically ordered set are also shown in the figure. Now consider pattern matches, with the target pattern '1'. In rows 1 to 4, the position numbers of source pattern strings satisfying the pattern match conditions are shown for the four possible target pattern positions within the source pattern string. For example, in row 4 the target pattern '1' occupies the first position in the source pattern string. This condition is satisfied for the binary numbers 1000 to 1111 , which have position numbers from 9 to 16. More generally, note that in a given row, the position numbers of source pattern strings where the target pattern occurs form one or several groups of successive numbers with breaks between groups. The number of elements in each group depends on the row, and hence the location of the target pattern within the source pattern string. In row 1 , each group contains a single element, with a gap of 1 between each group. In row 3, each group contains 4 elements, with a gap of 4 between each group.
As will become apparent, the elements of row 1 form an arithmetic series, where the starting element, b0, is equal to the position of the target pattern, Pp, and the difference is the number of elements in the alphabet set, N, raised to the power of the length of the target pattern (i.e. the number of elements in the target pattern), Lp:
bm = bo + (m-1 )NLp equation (1 )
We now define the highest position in each group as the group leader. Each row then has a group leader series. The group leader series of any row is related to the group leader series of an adjacent row by a factor of N, the number of elements in the alphabet set. We define the group leader series of row 1 as the root group leader series.
In figures 2 to 4, we present a second scenario. We begin with the alphabet set
{a,b,c,d,e} and a source data length, Lt, of 4. We wish to perform pattern matching, with the target pattern 'b'. The position number of the target pattern, Pp, is then 2. Figure 2 shows all the potential source pattern strings which contain the target pattern 'b'. The dotted lines indicate that not all source, pattern strings have been displayed in this area. Figure 3 corresponds exactly to figure 2, except that the source pattern strings are represented by their position numbers. Figure 4 corresponds to figure 3, except that a position number axis has been added, so that the position numbers are represented approximately to scale. From equation (1) above, the root group leader series can be represented by
bm = 2 + (m-1)51 ={2,7,12,17...} The group leader series of any of the rows is related to that of an adjacent row by a factor of N = 5.
The possible group leaders for source data length of 4 and all possible target patterns of length Lp = 1 are represented in table 3. Note that when the target pattern is 'b' (so Pp = 2) only groups having a group leader position of {2,7,12,...} represent a pattern match. The corresponding columns in the table are shaded.
Figure imgf000009_0001
Table 3 The pattern matching algorithm will now be described with reference to figures 3 and 5, in which the relevant step is indicated by bracketed numerals.
The algorithm searches for pattern matches at each possible target pattern position within the source pattern string, that is one row at a time. The number of rows is equal to 1 plus the difference in lengths between the source pattern string and the target pattern string, Lt - Lp +1 = LD + 1 , where LD = Lt - Lp (1). We therefore define Liter (2), which is incremented or decremented by 1 to move one row at a time.
We calculate the position numbers of the source and target pattern strings. If LD equals zero and Pt equals Pp , then the target pattern string is identical to the source pattern string (3). If so (4), then clearly we have one and only one pattern match (5) and the algorithm ends (6). Otherwise (7), we calculate the group leader position of the group including the source pattern string (8). This group leader position is then compared to the root group leader series (9). If the group leader position is not an element of the root group leader series (12) then processing proceeds to the next step (13). If the group leader position is an element of the root group leader series (10), then there is a pattern match in the corresponding row (11). If Lller does not equal zero (13,14) the algorithm loops through the remaining rows by decrementing L'ter (15), checking for pattern matches in each row. Once all of the rows have been checked, L'ter = 0 (13,16), the algorithm ends (17).
It is apparent that this algorithm searches for all pattern matches in the source data. However, a similar algorithm could easily search for a single pattern match, ending immediately after a pattern match is found. In this case, empirical knowledge of the strings involved could significantly speed up the algorithm. The sequence of searching may depend upon a characteristic of the source pattern string. If it were known a priori that the target pattern was likely to be contained towards the end of the source pattern string we should start with the assumption that the target pattern was contained in row 1 and move progressively one row at a time towards row (LD + 1 ). On the other hand, if the target pattern was likely to be contained towards the beginning of the source pattern string, we would move progressively from row (LD + 1) to row 1. Another similar algorithm could search for a pattern match at a particular position in the source pattern string. Alternatively the sequence of searching may be based upon an analysis of the source pattern string and occur according to the estimated probability for each row. It will be appreciated that a range of search sequences may be employed depending upon characteristics of the source data.
The step of calculating the group leader position, described above, can be achieved as follows. The position, Pj, of the lowest member of each possible group can be represented by:
Figure imgf000011_0001
where j={1 ,2,3...}, N is the number of elements in the alphabet and r is the row number.
Then
Figure imgf000011_0002
So that to find the group leader position, n, for a source pattern string of arbitrary position, Pt,
n = 1 + quotient (Pt- 1)/N
The step of determining whether group leader position, n, is an element of the root group leader series can be achieved using this equation:
R = remainder ((n - Pp)/NLp)
where Pp is the position of the target pattern within an ordered sequence of possible target patterns and Lp is the number of elements in the target pattern.
If R = 0, then the group leader position is an element of the root group leader series.
Example 1
An algorithm which performs the above steps is presented here in pseudo code. N - Number of elements in the alphabet. 7 - A source pattern string. P - A target pattern string. Lt - Length of T.
Lp - Length of P. pt - Position number of T Pp - Position number of P.
Rh - High value of the range of ac output for a particular string. Rι - Low value of the range of ac output for a particular string.
function Pattern_match
{ /* temporary variables 7 int Liter , z, d, n, m, temp; La = Lt Lp;
Liter = L ;
I* Ptδs Pp are calculated from equation (2) given below 7 If (Ld=0) and (Pt = Pp) I* equal length strings 7
MsgfBoth match"); Exit Function;
/* Two Arithmetic series are given by an = a0 + (n-1) and bm = b0 + (m-1)z 7 Loop for L + 1 times
{ find the source data's group leader position (n) in group arithmetic series 7 an = Pt; a0 = 1 d = power (N, Liter) temp = quotient ( (an - a0) / d); n = temp + 1 ;
/* check if 'n' falls in the root group leader series 7 bm = n ; bo = Pp ; ∑ = power (N, Lp); r = remainder ( (bm - b0)/z ); if (r = 0)
Msg ("pattern match success at position :" ((Ld + 1) - Liter));
L-iter = Ljter — 1 !
} /* Loop ends 7 We will now work through the algorithm using the example shown in figures 2 to 4. Consider a source pattern string input, such that the position of the source data P, » 158.
Firstly we look for a pattern match in Row 4.
Figure imgf000013_0001
I* find the source data's group leader position (n) in group arithmetic series 7
an-ϋPi'lSβ ao=1 d » power (N,L«*) = 53= 125 temp m quotient ( (a« - βo) d) = quotient ((158 -1 )125) = 1 ;
Figure imgf000013_0002
/* check if 'n* fails in the root group leader series 7
bm = n=:2. bo=P,>=2; z*ρowβr(N,l_p)«5. r s fβmabtdar ( O - hdffz ) » remainder ((2-2 yβ) = 0; r= therefors pattern match at poetlon 1
NØK! we look for a pattern match En Row 3. If
LIH.βLnl..1 =.2
Until Irso cssreo SMB grø teafer pcstisn (n) In grøp αrϊElwεtiG ossfeD 7 βaspjsta fl^asl «|α|MttBr0|,UH)a at25 temp » quotient ( (a« - βo) / d) - quotient ((158 -1)25) « 6; n =temp+1=7;
* check if 'n' falls in the root group leader series
bm = n = 7; boss p«2; z = power (N, L = 5; r = remainder ( ( m - øVz ) « remainder ((7-2)5) = 0; r=0 therefore pattern match at position 2
Next we look for a pattern match in Row 2.
Liter = Liter -1 = 1
/* find the source data's group leader position (n) in group arithmetic series 7
an = Pt = 158 a0 = 1 d = power (N, Liter) = 51 = 5 temp = quotient ( (an - a0) / d) = quotient ((158 -1 )/5) = 31 ; n = temp + 1 = 32 ;
/* check if 'n' falls in the root group leader series 7
bm = n = 32; b0 = Pp = 2; z = power (N, Lp) = 5; r = remainder ( (bm - b0)/z ) = remainder ((32-2)/5) = 0; r=0 therefore pattern match at position 3
Lastly we look for a pattern match in row 1.
i iter _. i iter Λ _ n
I" find the source data's group leader position (n) in group arithmetic series 7
an = Pt = 158 ao = 1 d = power (N, Liter) = 5° = 1 temp = quotient ( (an - a0) / d) = quotient ((158 -1 )/1 ) = 157; n = temp + 1 = 158 ;
/* check if 'n' falls in the root group leader series 7
bm = f = 158; bo = Pp = 2; z = power (N, Lp) = 5; r = remainder ( (bm - b0)/z ) = remainder ((158-2)/5) = 1; r ≠ 0 therefore NOT a pattern match at position 4 Where the target pattern consists of multiple characters the same algorithm may be employed.
In arithmetic codes, a string is represented by an interval on a number line. The size of the interval is determined by the probabilities of the symbols of the alphabet. In the equiprobable case, it is a simple matter to calculate the position number of a string:
P = Rh/(Rh - Rt) equation (2)
where Rh is the highest value of the interval and Rι is the lowest value of the interval.
In the case where different elements of the alphabet have different probabilities, the position number can still be determined, although this is not as straightforward as in the equiprobable case. We take into account the probabilities of each element of the alphabet set and the length of the source pattern to establish the position number that corresponds to the interval of interest. So, when operating on arithmetically compressed data, the position number of a coded interval is calculated. This position number, together with the target pattern's position number, becomes the input to the above algorithm.
Example 2
The source sequence can be represented as a position in a list of all possible combinations of the elements of the alphabet with the length of the source sequence - the source lexicon. The source sequence itself is in fact that position when represented in base n (n is the number of elements in the alphabet) where the elements of the alphabet represent digits in base n.
A set of the positions of all sub-sequences of the source sequence, which include the first character position of the source sequence, within a lexicon of all possible combinations of the elements of the alphabet with the length of the sub-sequence, can be computed. The set will contain the following elements: the first digit of the source sequence in base n, the first digit and the second digit, the first digit and the second and the third, and so on for the length of the source sequence position in base n. Within the source lexicon, "words" containing the target sequence are n to the power of the length of the target sequence apart. Using the position of the first "word" containing the target sequence, a series of positions of "words" containing the target sequence can therefore be defined.
The set and the series needed to be rendered in the same base and then a correlation procedure can be run to see if any of the possible target sequence positions are equivalent to the source sub-sequence positions.
If the set is ordered from smallest element to largest, the position of the element in the set which also matches with the target t series describee the position in the source sequence of the rightmost element of the target sequence.
Of course the βet of source sub-sequence positions may be matched against the series of possible target positions in O(n) by using the remainder equation.
Thus it may be determined whether a target sequence tX....Xm] is present within a eβureβ sβquørø© [Yι...YJ u βrβ {Xι..._ an |Yι...Yq} ar© members of a finite ordered t«t{2....-5} tey: i. constructing a set of source subsequence values in base o
{tVιJ,.-.|Vι.»V&..-Itfι..-Vlj} wrfiβre Vt a i : Yk a Zjj a l ii. determining whether any values of the set correspond to a series of target values {p,,..p+ko ,...p+ιτom} where p is the number [D1...Df)] in base o such tea Pt. = h : ^ = _ζ
T© dttermfπβ whether any values f the sit ccrreβpβnd to the series in step El the rest-i tor the following algorithm must be zero for that value of the set:
R = remainder ((N - p)/om)
Where:
N * a member of the set in base 10 The method can be implemented using a range of standard data processing devices or specialised graphics audio processing or sequencing devices. For example the method may be deployed on a personal computer operating under the MICROSOFT WINDOWS™ environment or on a server machine operating under the UNIX operating system.
It will be appreciated that the method may be implemented in software executing on a device or implemented in specialised hardware.
It will be further appreciated by those skilled in the art that method may be deployed in portions over a multiple of computers or servers.
It is believed that the present invention represents the first method for pattern matching of arithmetically compressed source data without decompression of the data. The invention can also operate using a compressed target pattern string. The invention should be of immense utility at arithmetic decoder stations, where the decoder could typically check for partial or full matches with any prior strings of data. The invention can be used to find all pattern matches, one pattern match or a pattern match at a particular position in the source pattern string. The invention can also be executed in parallel, as a search on a particular interval can be performed independently of searches on other intervals. That is, the invention can be applied to more than one source string at the same time. The invention could also operate in a decentralised system, for example a search engine accessible over a communications network.
While the present invention has been illustrated by the description of the embodiments thereof, and while the embodiments have been described in detail, it is not the intention of the Applicant to restrict or in any way limit the scope of the appended claims to such detail. Additional advantages and modifications will readily appear to those skilled in the art. Therefore, the invention in its broader aspects is not limited to the specific details, representative apparatus and method, and illustrative examples shown and described. Accordingly, departures may be made from such details without departure from the spirit or scope of the Applicant's general inventive concept.

Claims

Claims:
1. A method of determining whether a target pattern is present within a source pattern composed of one or more characters from an alphabet set by determining whether the position of the source pattern within a sequence of possible patterns is a position which correlates with a position within the sequence of possible patterns that includes the target pattern.
2. A method as claimed in claim 1 comprising the steps of: i. identifying the group leader for a source pattern where the group leader is the highest position number in each group formed by dividing a sequence of possible position numbers into groups according to the base of the alphabet system; and ii. determining whether the group leader value of the source pattern is a group leader value that correlates with the position of the target pattern within the sequence of possible patterns that includes the target pattern
3. A method as claimed in claim 2 wherein the group leader value of the source pattern is determined directly or indirectly for each row by dividing an ascending sequence of possible position numbers into groups having a group size corresponding to the base of the alphabet raised to the power of the row less one, where the row corresponds to a position of the target pattern within the source pattern.
4. A method as claimed in claim 3 wherein a group leader position is used to determine whether the group leader value is valid for the target pattern and where the group leader position is the position of the group leader's group amongst the groups for that row.
5. A method as claimed in any one of claims 2 wherein the group leader position for the source pattern is calculated according to the following algorithm:
N = 1 + Quotient((Sp - W1) Where:
N= source pattern group leader position number
Sp = Source pattern position number b = alphabet base r = row number
6. A method as claimed in claim 5 wherein to determine whether there is a match in step ii the result for the following algorithm must be zero:
R = remainder ((N - Tp)/bτι)
Where:
R = the remainder Tp = the target pattern position number
Tl = the length of the target pattern
7. A method as claimed in claim 3 wherein there are a number of rows and the method is applied consecutively to each row.
8. A method as claimed in claim 3 wherein the method is applied to more than one source pattern simultaneously.
9. A method as claimed in claim 3 wherein the order in which the rows are searched is dependent on a characteristic of the source pattern.
10. A method as claimed in claim 3 wherein the probability of finding the target pattern in a row is estimated and is used to determine the order in which the rows are searched.
11. A method as claimed in claim 3 wherein the probability of finding the target pattern in a row is known and is used to determine the order in which the rows are searched.
12. A method as claimed in claim 1 wherein the source pattern is arithmetic coded data.
13. A method as claimed in claim 12 wherein the target pattern is arithmetic coded data.
14. A method as claimed in claim 12 wherein the source pattern is compressed using arithmetic coding.
15. A method as claimed in claim 13 wherein the target pattern is compressed using arithmetic coding.
16. A method as claimed in claiml wherein the target pattern has more than one element.
17. A method of determining whether a target sequence [X1...Xm] is present within a source sequence [Y1...Yq] where {X1...Xm} and {Y1...Yq} are members of a finite ordered set {Z1...Zo}, including the steps of: i. constructing a set of source sub-sequence values in base o {rV1],...r 1...Vk],...r 1...Vn]}where Vk = j : Yk = Zj; and ii. determining whether any values of the set correspond to a series of target values {p,...p+kom,...p÷nom} where p is the number [D^.-Dn] in base o
Figure imgf000020_0001
18. A method as claimed in claim 17 wherein to determine whether any values of the set correspond to the series in step ii the result for the following algorithm must be zero for all members of the set:
R = remainder ((N - p)/om)
Where:
N = a member of the set in base 10
19. A method as claimed in claim 17 wherein the source sequence is image data.
20. A method as claimed in claim 17 wherein the source sequence is video data.
21. A method as claimed in claim 17 wherein the source sequence is bio-medical data.
22. A data processing apparatus to determine whether a target pattern is present within a source pattern composed of one or more characters of an alphabet, comprising: i. a first memory for storing a target pattern; a second memory for storing a source pattern; a processing means for determining the position of the source pattern in a sequence of possible patterns; iv. a processing means for determining the position of the target pattern in the sequence of possible patterns; and v. a processing means for correlating the source position with the target position.
23. A data processing apparatus to determine whether a target sequence is present within a source sequence composed of one or more characters of an alphabet, comprising: i. a first memory for storing a target sequence position wherein the target sequence position is the position of the target sequence within a lexicon of all possible combinations of characters of the alphabet; ii. a second memory for storing a source sequence position wherein the source sequence position is the position of the source sequence within the lexicon; iii. a processing means for computing a set of positions of sub-sequences of the source sequence wherein the position of the sub-sequence is the position of the sub-sequence within the lexicon and wherein the subsequence includes the first character position of the source sequence; iv. a processing means for determining a series defining all positions of sequences within the lexicon which contain the target sequence; v. a processing means for correlating the set with the series.
24. A data processing apparatus as claimed in claim 22 wherein the source sequence is image data.
25. A data processing apparatus as claimed in claim 22 wherein the source sequence is video data.
26. A data processing apparatus as claimed in claim 22 wherein the source sequence is bio-medical data.
27. A data processing apparatus as claimed in claim 22 wherein the processing means is a computer system.
28. A data processing apparatus adapted to operate in accordance with the method of claim 1.
29. A data processing apparatus adapted to operate in accordance with the method of claim 17.
30. A computer program which performs at the method of claim 1.
31. A computer program which performs at the method of claim 17.
32. Computer readable medium embodying a computer program as claimed in claim 30.
33. Computer readable medium embodying a computer program as claimed in claim 31.
34. A signal embodying a computer program as claimed in claim 30.
35. Computer readable medium embodying a computer program as claimed in claim 31.
36. A computer programmed to operate in accordance with the method of claim 1.
37. A computer programmed to operate in accordance with the method of claim 17.
38. Pattern matching information produced by the method of claim 1.
39. Pattern matching information produced by the method of claim 17.
40. A signal embodying pattern matching information as claimed in claim 38.
41. A signal embodying pattern matching information as claimed in cfaim 39.
PCT/IN2004/000059 2003-03-13 2004-03-12 A method and system for pattern matching WO2004081819A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP04720131A EP1602041A1 (en) 2003-03-13 2004-03-12 A method and system for pattern matching
JP2006507626A JP2006522401A (en) 2003-03-13 2004-03-12 Pattern matching method and system

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US10/386,462 2003-03-13
US10/386,462 US7840072B2 (en) 2003-03-13 2003-03-13 Method and system for pattern matching

Publications (1)

Publication Number Publication Date
WO2004081819A1 true WO2004081819A1 (en) 2004-09-23

Family

ID=32987317

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IN2004/000059 WO2004081819A1 (en) 2003-03-13 2004-03-12 A method and system for pattern matching

Country Status (5)

Country Link
US (1) US7840072B2 (en)
EP (1) EP1602041A1 (en)
JP (1) JP2006522401A (en)
CN (1) CN1759394A (en)
WO (1) WO2004081819A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007116549A (en) * 2005-10-21 2007-05-10 Mitsubishi Electric Corp Network relay unit

Families Citing this family (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TW530495B (en) * 2001-05-16 2003-05-01 Cyberlink Corp System and method for processing compressed data stream
US7810155B1 (en) * 2005-03-30 2010-10-05 Symantec Corporation Performance enhancement for signature based pattern matching
US8930431B2 (en) 2010-12-15 2015-01-06 International Business Machines Corporation Parallel computation of a remainder by division of a sequence of bytes
US20150302050A1 (en) * 2012-05-24 2015-10-22 Iqser Ip Ag Generation of requests to a data processing system
CN103873317B (en) * 2012-12-18 2017-04-12 中国科学院空间科学与应用研究中心 Method and system for detecting CCSDS (consultative committee for space data system) space link protocol
CN104252469B (en) 2013-06-27 2017-10-20 国际商业机器公司 Method, equipment and circuit for pattern match
WO2015132914A1 (en) * 2014-03-05 2015-09-11 三菱電機株式会社 Data compression apparatus and data compression method
KR101595189B1 (en) * 2014-11-14 2016-02-19 인하대학교 산학협력단 A method of pattern matching on compressed texts based on boyer-moore-horspool algorithm
US9959299B2 (en) 2014-12-02 2018-05-01 International Business Machines Corporation Compression-aware partial sort of streaming columnar data
CN105893337B (en) * 2015-01-04 2020-07-10 伊姆西Ip控股有限责任公司 Method and apparatus for text compression and decompression
US10909078B2 (en) * 2015-02-25 2021-02-02 International Business Machines Corporation Query predicate evaluation and computation for hierarchically compressed data
US9941004B2 (en) 2015-12-30 2018-04-10 International Business Machines Corporation Integrated arming switch and arming switch activation layer for secure memory

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4122440A (en) 1977-03-04 1978-10-24 International Business Machines Corporation Method and means for arithmetic string coding
US5936559A (en) * 1997-06-09 1999-08-10 At&T Corporation Method for optimizing data compression and throughput

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
BELL ET AL: "Pattern Matching in Compressed Text and Images", TECHNICAL REPORTS, TR-COSC, no. 07/01, 29 May 2001 (2001-05-29), UNIVERSITY OF CANTERBURY, XP002292206, Retrieved from the Internet <URL:HTTP://WWW.COSC.CANTERBURY.AC.NZ/RESEARCH/REPORTS/TECHREPS/2001/TR_0107.PDF> [retrieved on 20040810] *
LANGDON G: "An introduction to arithmetic coding", IBM JOURNAL OF RESEARCH AND DEVELOPMENT, IBM CORPORATION, ARMONK, US, vol. 28, no. 2, 1 March 1984 (1984-03-01), pages 135 - 149, XP002086019, ISSN: 0018-8646 *
WITTEN I H ET AL: "ARITHMETIC CODING FOR DATA COMPRESSION", COMMUNICATIONS OF THE ASSOCIATION FOR COMPUTING MACHINERY, ASSOCIATION FOR COMPUTING MACHINERY. NEW YORK, US, vol. 30, no. 6, 1 June 1987 (1987-06-01), pages 520 - 540, XP000615171, ISSN: 0001-0782 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007116549A (en) * 2005-10-21 2007-05-10 Mitsubishi Electric Corp Network relay unit
JP4627243B2 (en) * 2005-10-21 2011-02-09 三菱電機株式会社 Network relay device

Also Published As

Publication number Publication date
JP2006522401A (en) 2006-09-28
CN1759394A (en) 2006-04-12
US7840072B2 (en) 2010-11-23
EP1602041A1 (en) 2005-12-07
US20040199931A1 (en) 2004-10-07

Similar Documents

Publication Publication Date Title
CN109635273B (en) Text keyword extraction method, device, equipment and storage medium
EP0510634B1 (en) Data base retrieval system
US7454431B2 (en) Method and apparatus for window matching in delta compressors
Gueniche et al. Compact prediction tree: A lossless model for accurate sequence prediction
KR100414236B1 (en) A search system and method for retrieval of data
US8908978B2 (en) Signature representation of data having high dimensionality
WO2004081819A1 (en) A method and system for pattern matching
Gawrychowski Pattern matching in Lempel-Ziv compressed strings: fast, simple, and deterministic
US6542644B1 (en) Statistical data compression/decompression method
CN109299235B (en) Knowledge base searching method, device and computer readable storage medium
US9069634B2 (en) Signature representation of data with aliasing across synonyms
Amir et al. Repetition detection in a dynamic string
Boffa et al. A learned approach to design compressed rank/select data structures
Boffa et al. A “Learned” Approach to Quicken and Compress Rank/Select Dictionaries∗
Puglisi et al. Data compression and learning in time sequences analysis
CN114567332A (en) Text secondary compression method, device and equipment and computer readable storage medium
CN110674635B (en) Method and device for dividing text paragraphs
Bell et al. Searching BWT compressed text with the Boyer-Moore algorithm and binary search
JP3545007B2 (en) Database search system
CN111538803A (en) Method, device, equipment and medium for acquiring candidate question text to be matched
Ehlers et al. k-Abelian pattern matching
Lysyak et al. Time series prediction based on data compression methods
CN114238564A (en) Information retrieval method and device, electronic equipment and storage medium
JP2993540B2 (en) Ascending integer sequence data compression and decoding system
Gagie et al. Space-efficient conversions from SLPs

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BW BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE EG ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NA NI NO NZ OM PG PH PL PT RO RU SC SD SE SG SK SL SY TJ TM TN TR TT TZ UA UG US UZ VC VN YU ZA ZM ZW

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): BW GH GM KE LS MW MZ SD SL SZ TZ UG ZM ZW AM AZ BY KG KZ MD RU TJ TM AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IT LU MC NL PL PT RO SE SI SK TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG

121 Ep: the epo has been informed by wipo that ep was designated in this application
DPEN Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed from 20040101)
WWE Wipo information: entry into national phase

Ref document number: 2004720131

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 1727/KOLNP/2005

Country of ref document: IN

WWE Wipo information: entry into national phase

Ref document number: 20048065532

Country of ref document: CN

Ref document number: 2006507626

Country of ref document: JP

WWP Wipo information: published in national office

Ref document number: 2004720131

Country of ref document: EP