JPWO2004088539A1 - Parsing method and programmable processor unit for parsing - Google Patents

Abstract

In the syntax analysis method according to the present invention, a table including a plurality of sets of tables each including a plurality of sets of one set of reference data corresponding to a keyword, reference destination table data corresponding to the reference data, and flag data is stored in a memory. A step of preparing a plurality corresponding to the address data, a step of extracting one table from the memory, a step of extracting target data from the stream data, and each reference of the target data and a plurality of sets of data included in the extracted table A step of comparing the data, a step of selecting reference destination table data and flag data corresponding to the reference data matching the target data, and a reference destination selected from the memory using the selected reference destination table data as address data Corresponding to table data And a step of outputting the Buru.

Description

  The present invention relates to a parsing method for finding a desired keyword from media data such as MPEG, stream data such as multiplexed binary stream data and character strings, and the syntax that can be processed particularly at high speed. Use the analysis method. The invention further relates to a programmable processor unit for such parsing.

In the conventional syntax analysis, one keyword to be searched is taken out, and the keyword and stream data to be searched are compared from the beginning to the end of the stream data. When the comparison between one keyword and stream data is completed, the process is repeated such that the next comparison is performed for the next keyword.
The procedure of the conventional syntax analysis method will be described with reference to FIG. In FIG. 13, it is assumed that stream data 1300 (cabcdea...) To be searched is stored in the buffer. Also, it is assumed that the keyword 1310 (“abcd”, “cad”, “cae”, and “cd”) to be searched is also stored in the buffer. First, the keyword “abcd” is extracted, and as shown in the first search 1320, the comparison is performed in order from the head of the stream data 1300. In the example of FIG. 13, the second to fifth characters of the stream data 1300 match the keyword “abcd”. The comparison is performed from the sixth character to the end of the stream data 1300 while holding the match by some means. In this way, the first search 1320 is completed.
Next, the keyword “cad” is taken out and compared as shown in the second search 1330 in order from the top of the stream data 1300. Further, the comparison is continued by the number of keywords 1310 in the order of the third search 1340 and the fourth search 1350, and all search processes are completed.
As a technique for performing such comparison at high speed, there is a hash method. The hash method is a method in which a function (hash function) for obtaining an address of a storage location of an element is defined from elements of a set of character strings, and the address obtained by the hash function is stored in a table. A conventional example using the hash method is described in Japanese Patent Laid-Open No. 4-96174. In the conventional example, the candidate character string reading means and the match search means are configured to operate independently to increase the speed of comparison.
However, the conventional parsing has a problem that the search time increases in proportion to the number of keywords when there are a plurality of keywords to be compared. There is also a problem that the parsing cannot be started unless all the stream data to be searched is prepared. Furthermore, there is a problem that keywords are usually fixed and set and cannot be easily changed.
SUMMARY OF THE INVENTION Therefore, the present invention compares all the keywords registered in advance and developed in the table with the target data extracted from the stream data in parallel. It is an object of the present invention to provide a parsing method and a programmable processor unit for such parsing.
It is another object of the present invention to provide a syntax analysis method capable of performing a search even when all the stream data to be searched is not available, and a programmable processor unit for such syntax analysis.
In order to achieve the above object, a syntax analysis method according to the present invention includes a plurality of sets of data each composed of reference data corresponding to a keyword, reference destination table data corresponding to the reference data, and flag data in a memory. A step of preparing a plurality of tables corresponding to address data, a step of extracting one table from a memory, a step of extracting target data from stream data, and a plurality of sets of data included in the extracted table of target data A step of comparing each of the reference data, a step of selecting reference destination table data and flag data corresponding to the reference data that matches the target data, and selecting from the memory using the selected reference destination table data as address data For the referenced table data A process of outputting the table, and by repeating the extraction process, the comparison process, the selection process, and the output process while sequentially extracting the data constituting the stream data as new target data, keywords are extracted from the stream data. It is characterized by detecting.
In the parsing method according to the present invention, each piece of data is extracted as target data from the stream data and sequentially compared with keywords expanded in a table on the memory. Even if not, syntax analysis can be started only by the amount existing from the beginning of the stream data. Moreover, even if stream data to be searched is added in the middle, parsing can be easily resumed or continued. Furthermore, even when there are a plurality of keywords to be searched, by searching all the keywords on the memory, a search for all the keywords can be processed in parallel.
In order to achieve the above object, a programmable processor unit according to the present invention includes a register for storing target data extracted from stream data, reference data corresponding to a keyword, and a reference table corresponding to the reference data. A plurality of table sets each including a plurality of sets of data and flag data, a memory that outputs a table set corresponding to the address data in response to input of the address data, and one table set from the memory A plurality of sets of data is input, the target data is compared with each reference data of the plurality of sets of data, and if they match, the reference table data and flag data corresponding to the matched reference data are selected, Selected reference table data The table set corresponding to the input from memory, and having a processing circuit for outputting a referenced table data selected as the address data.
The programmable processor unit according to the present invention is a unit for realizing the syntax analysis method according to the present invention described above. Further, since the keyword to be searched can be set as a program, the keyword can be easily changed according to the specifications of the incoming stream data. Further, even when there are a plurality of keywords to be searched, it is possible to process searching for all keywords simultaneously.

FIG. 1 is a flowchart showing the overall procedure of the syntax analysis method according to the present invention.
FIG. 2 is a diagram for explaining a procedure for expanding a keyword table.
FIG. 3 is a diagram illustrating an example of data developed in a table.
FIG. 4 is a diagram for explaining an outline of a procedure for searching a data stream.
FIG. 5 is a diagram illustrating an example of data stream search.
FIG. 6 is a diagram illustrating an example of data stream search.
FIG. 7 is a diagram illustrating an example of data stream search.
FIG. 8 is a diagram illustrating an example of data stream search.
FIG. 9 is a diagram illustrating an example of data stream search.
FIG. 10 is a diagram illustrating an example of a table stored in the memory.
FIG. 11 is a diagram showing a schematic configuration of a programmable processor unit according to the present invention.
FIG. 12 is a diagram showing a schematic circuit configuration of the processing circuit shown in FIG.
FIG. 13 is a diagram for explaining a conventional syntax analysis method.

FIG. 1 is a flowchart showing a procedure of a syntax analysis method according to the present invention. The syntax analysis method according to the present invention is configured so that a keyword expanded in a table in advance is searched from a data stream to be searched, and a search result is output.
The following is an example in which a search is performed for four types of keywords (“abcd”, “cad”, “cae”, and “cd”) using a character string that uses the following eight characters as a data stream: I will explain.
Character code data
0000 a
0001 b
0002 c
0003 d
0004 e
0005 f
0006 g
0007 h
First, the total number of tables and the depth of the pointer array are defined, and the table is initialized (step 101). Here, the total number of tables is set to 6, and the depth N of the pointer array is set to 10.
Next, the total number of keywords to be registered is defined, and keyword initialization is performed (step 102). Here, the total number of keywords is set to 4.
Next, keywords to be searched are registered one by one and developed in a table (step 103). FIG. 2 shows an example in which the keyword “cae” is registered and table expansion is performed.
In FIG. 2, it is assumed that the keyword “abcd” has already been registered. By registering the keyword “abcd”, the first character table 200, the second character table 210, the third character table 220, and the fourth character table 230 are created. Each table corresponds to address data as illustrated, and includes a plurality of sets of data including reference data, reference table data, and flag data. Further, the reference data corresponds to eight types of characters constituting the character string to be searched. In each table, the initial value of the reference table data is set to [0x0000], and the initial value of the flag data is set to [unmatch].
First, in the first character table 200, the reference table data corresponding to the reference data “c” is changed to [0x0004] according to the first character of the keyword “cae”, and the flag data corresponding to the reference data “c” is changed. Change to [continue].
Next, 240 is newly created as the second character table corresponding to the second character of the keyword “cae” and the reference table data [0x0004]. Further, the reference table data corresponding to the reference data “a” in the second table 240 is changed to [0x0005], and the flag data corresponding to the reference data “a” is changed to [continue].
Next, 250 is newly created as the third character table corresponding to the third character of the keyword “cae” and the reference table data [0x0005]. Further, the flag data corresponding to the reference data “e” in the third table 250 is changed to [match], and the registration of the keyword “cae” is completed.
Similarly, the keywords “cae” and “cd” are registered. A plurality of table sets after registration of all keywords and table expansion are shown in FIG. The plurality of table sets shown in FIG. 3 includes a first character table 300, a second character table 310 and 340, a third character table 320 and 340, and a fourth character table 350.
Next, a plurality of table sets are displayed on an appropriate display after registration of all keywords and table expansion are completed (step 104). The display of the plurality of table sets is for debugging, and can be omitted if unnecessary.
Next, a keyword search is executed from the character string to be searched (step 105). FIG. 4 shows an outline of the search procedure. Here, it is assumed that the character string to be searched is “cabcde...”. First, the first character “c” is extracted as target data from the search target character string, and “c” is compared with the first character of the four keywords (401). Since the first character of the target data “c” and the keywords “cad”, “cae”, and “cd” match, these keywords are assumed to have continued.
Next, the next character “a” of the character string to be searched is extracted as target data, and “a” is compared with the four keywords (402). Since the target data “a” and the second characters of the keywords “cad” and “cae” match, it is assumed that these keywords have continued. Further, since the second character of the keyword “cd” is not “a”, they do not match. Further, since the first character of the target data “a” and the keyword “abcd” match, this keyword is assumed to be continued.
Next, the next character “b” of the search target character string is extracted as target data, and “b” is compared with the four keywords (403). Since the second character of the target data “b” and the keyword “abcd” match, it is assumed that the keyword continues. Further, since the third characters of the keywords “cad” and “cae” are not “b”, they do not match.
Next, the next character “c” of the character string to be searched is extracted as target data, and “c” is compared with the four keywords (404). Since the third character of the target data “c” and the keyword “abcd” match, this keyword is assumed to be continued. Furthermore, since the target data “c” and the first characters of the keywords “cad”, “cae”, and “cd” match, it is assumed that these keywords continue.
Next, the next character “d” of the character string to be searched is extracted as target data, and “d” is compared with the four keywords (405). Since the fourth character of the target data “d” and the keyword “abcd” match, it is assumed that this keyword is detected in the character string. Further, since the second characters of the keywords “cad” and “cae” are not “d”, these keywords do not match. Further, the second character of the keyword “cd” is “d”, but since the keyword “abcd” matches, the keyword “cd” is invalidated.
In this way, the process of extracting characters one by one from the search target character string and performing comparison with all the keywords simultaneously is performed for all the search target character strings.
Next, using FIG. 5 to FIG. 9, how the above-described search process is executed using a plurality of table sets and pointers will be described.
First, “c” is extracted as target data from the character string to be searched for comparison (see FIG. 5). First, the table 300 is selected based on [0x0000] initially set in the 0th hierarchy of the pointer (501), the target data “c” is compared with the reference data “a” to “f”, and the matched reference is selected. Since the flag data of the data “c” is [continue], the reference table data [0x0004] is stored in the pointer (502). Therefore, after the comparison with the first target data “c”, the pointer is set like a pointer (rear) 370. [0x0000] is always set in the lowest layer of the pointer.
Next, the next target data “a” is extracted from the character string to be searched and compared (see FIG. 6). At this time, the pointer is set as a pointer (previous) 360 from the result of FIG. First. The table 340 is selected based on [0x0004] set in the 0th layer of the pointer (601), the target data “a” is compared with the reference data “a” to “f”, and the matched reference data “ Since the flag data of “a” is [continue], the reference table data [0x0005] is stored in the pointer (602). Next, the table 300 is selected based on [0x0000] set in the first layer of the pointer (603), the target data “a” is compared with the reference data “a” to “f”, and they match. Since the flag data of the reference data “a” is [continue], reference destination table data [0x0001] is stored in the pointer (604). Therefore, after the comparison with the target data “a”, the pointer is set as a pointer (after) 370.
Next, the next target data “b” is extracted from the character string to be searched and compared (see FIG. 7). At this time, the pointer is set as a pointer (front) 360 from the result of FIG. First. The table 350 is selected based on [0x0005] set in the 0th hierarchy of the pointer (701), the target data “b” is compared with the reference data “a” to “f”, and the matched reference data “ Since the flag data of “b” is [unmatch], it is not stored in the pointer. Next, the table 310 is selected based on [0x0001] set in the first layer of the pointer (702), the target data “b” is compared with the reference data “a” to “f”, and they match. Since the flag data of the reference data “a” is [continue], the reference table data [0x0002] is stored in the pointer (703). Next, the table 300 is selected based on [0x0000] set in the second layer of the pointer (704), the target data “b” is compared with the reference data “a” to “f”, and they match. Since the flag data of the reference data “b” is “unmatch”, it is not stored in the pointer. Therefore, after the comparison with the target data “b”, the pointer is set as a pointer (rear) 370.
Next, the next target data “c” is extracted from the character string to be searched and compared (see FIG. 8). At this time, the pointer is set as a pointer (previous) 360 from the result of FIG. First, the table 320 is selected based on [0x0002] set in the 0th hierarchy of the pointer (801), the target data “c” is compared with the reference data “a” to “f”, and the matched reference is obtained. Since the flag data of the data “c” is [continue], the reference table data [0x0003] is stored in the pointer (802). Next, the table 300 is selected based on [0x0000] set in the first layer of the pointer (803), the target data “c” is compared with the reference data “a” to “f”, and they match. Since the flag data of the reference data “c” is [continue], the reference destination table data [0x0004] is stored in the pointer (804). Therefore, after the comparison with the target data “c”, the pointer is set as a pointer (rear) 370.
Next, the next target data “d” is extracted from the search target character string and compared (see FIG. 9). At this time, the pointer is set as a pointer (previous) 360 from the result of FIG. First. The table 330 is selected based on [0x0003] set in the 0th layer of the pointer (901), the target data “d” is compared with the reference data “a” to “f”, and the matched reference data “ Since the flag data of “d” is [match], it is determined that the keyword “abcd” has been detected. Next, the table 340 is selected based on [0x0004] set in the first layer of the pointer (902), the target data “d” is compared with the reference data “a” to “f”, and they match. Since the flag data of the reference data “d” is [match] but the keyword “abcd” is detected at the same level, the detection of the keyword “cd” is invalidated. Next, the table 300 is selected based on [0x0000] set in the second layer of the pointer (903), the target data “d” is compared with the reference data “a” to “f”, and they match. Since the flag data of the reference data “d” is “unmatch”, it is not stored in the pointer. Therefore, after the comparison with the target data “d”, the pointer is set as a pointer (rear) 370.
In this manner, the target data is extracted character by character from the character string to be searched, and the comparison with the keyword is performed up to the end of the character string to be searched. Therefore, in the syntax analysis method according to the present invention, even when the character strings to be searched are not prepared to the end, the syntax analysis can be started only by the amount existing from the top of the character string. Even if a search target character string is added in the middle, parsing can be easily resumed or continued. Furthermore, even when there are a plurality of keywords to be searched, it is possible to process all keyword searches in parallel.
Next, a programmable processor unit for efficiently realizing the syntax analysis method according to the present invention described above will be described with reference to FIGS.
FIG. 10 shows an outline of a programmable processor unit 1000 according to the present invention. The programmable processor unit 1000 shown in FIG. 10 is an example, and the present invention is not limited to this. As shown in the figure, the programmable processor unit 1000 includes a processing circuit 1002, a target register 1004, a delay circuit 1006, and a memory 1008.
In the programmable processor unit 1000, the character string 1010 to be searched is input to the target register 1004, and is input to the processing circuit 1002 as target data 1012 character by character. The processing circuit 1002 outputs a NEXT address signal 1014 to be temporarily stored in the delay circuit 1006. An address signal 1016 is input to the memory 1008 at a predetermined timing. A plurality of table sets to be described later are stored in advance in the memory 1008, and a plurality of sets of data 1018 (reference data [n], reference destination table data [n], valid byte data [ n] and flag data [n]) are input to the processing circuit 1002. The processing circuit 1002 is configured to output the NEXT address signal 1014 and the flag data 1020 using the target data 1012 and a plurality of sets of data 1018 included in the table set according to the theory based on the syntax analysis method described above. .
Next, a plurality of table sets stored in the memory 1008 will be described. The plurality of table sets may be six table sets 300 to 350 as shown in FIG. 3 described above. However, it is also possible to create a table set as shown in FIG. 11 by compressing items that are not used. In FIG. 11, an example table set 1100 includes address data 1102, reference data 1104, reference destination table data 1106, valid byte data 1108, and flag data 1110. The valid byte data 1108 is data indicating whether or not comparison of corresponding reference data is necessary.
If the table set shown in FIG. 11 is used, four sets of set data (reference data [n], reference destination table data [n], valid byte data [n], and flag data [n]) to be processed at a time are used. And the hardware configuration can be simplified.
Next, an outline of a circuit configuration of the processing circuit 1002 is shown in FIG. The processing circuit 1002 shown in FIG. 12 is suitable when the table set 1100 shown in FIG. 11 is used, and can process four sets of data at a time.
As shown in FIG. 12, the main part of the processing circuit 1002 includes an address decoder 1200, comparators 1202, 1208, 1214 and 1220, inverters 1204, 1210, 1216 and 1222, OR circuits 1206, 1212, 1218 and 1224, and a priority encoder. 1226, a first output buffer 1228, and a second output buffer 1230. The processing circuit 1002 is configured based on the theory of simultaneously comparing four reference data when the effective byte is 1 bit.
The processing circuit 1002 refers to four sets of data (reference data [n], reference destination table data [n], valid byte data [n], and flag data [n]) corresponding to the address data from the memory 1008 at the same time. In FIG. 12, the four sets of referenced data are shown as 1240.
The reference data [0] to [3] of each row is decoded by the address decoder 1200 and input to the comparators 1202, 1208, 1214, and 1220, and is compared with the target data. When the decoded reference data matches the target data, a high level is output from the comparator.
Valid byte data [0] to [3] of each row are decoded and input to inverters 1204, 1210, 1216 and 1222, respectively. Therefore, when the valid byte data is “valid”, a low bell is output from each inverter, and when the valid byte data is “invalid”, a high bell is output from each inverter.
Each reference table data [0] to [3] is stored in the first output buffer 1228, and each flag data [0] to [3] is stored in the second output buffer 1230.
When the inverter output is low level, the OR circuit outputs a high level when the output from the comparator is high level. That is, in this case, the decoded reference data matches the target data. Further, when the inverter output is at a high level, the OR circuit outputs a high level unconditionally.
The priority decoder 1226 outputs the reference destination table data of the set from which the high level is output from the OR circuit as the next address data 1014 from the first output buffer 1228, and similarly the set from which the high level is output from the OR circuit. Are output as flag data 1020 from the second output buffer.
The next address data 1014 output from the first output buffer is temporarily held by the delay circuit 1006 and waits for the next sequence. In the next sequence, the next target data is updated and the same processing is performed. The sequence is repeated until the processing for all the character strings is completed. In this way, a search is performed to determine whether or not a keyword exists for all character strings.
As described above, it is possible to search at high speed whether there is a keyword previously expanded in the memory 1008 in the stream data. In particular, if this programmable processor unit is used, even if the stream data to be searched is not complete to the end, it is possible to start parsing only by the amount existing from the beginning. Furthermore, parsing can be easily resumed or continued even if the data stream is added midway. Furthermore, even when there are a plurality of keywords to be searched, it is possible to process all keyword searches in parallel. Therefore, when stream data in which image data such as MPEG and control data are mixed is sequentially received using communication means, the present invention is particularly useful when searching for only control data from such stream data. It is valid.
In addition, since keywords can be easily programmed in the memory, a plurality of groups of keywords are expanded in a table and stored in the memory in advance, and the keyword to be searched is switched according to the incoming stream data. Is also possible.

Claims (7)

In the syntax analysis method for detecting keywords from stream data to be searched,
A plurality of tables corresponding to the address data are prepared in the memory in correspondence with the address data. The table includes a plurality of sets of one set of reference data corresponding to the keyword, reference destination table data corresponding to the reference data, and flag data. Process,
Removing one table from the memory;
Extracting target data from the stream data;
A step of comparing the target data with each reference data of a plurality of sets of data included in the extracted table;
Selecting reference table data and flag data corresponding to reference data that matches the target data;
Using the selected reference table data as address data, and outputting a table corresponding to the selected reference table data from the memory,
Detecting the keyword from the stream data by repeatedly performing the extraction step, the comparison step, the selection step, and the output step while sequentially extracting data constituting the stream data as new target data. Character parsing method. The syntax analysis method according to claim 1, wherein the flag data indicates whether or not the keyword is included in the stream data. The syntax analysis method according to claim 1, wherein the memory includes a set of data including reference data, flag data, and reference destination table data corresponding to a plurality of keywords. In a programmable processor unit for detecting programmable keywords from stream data,
A register for storing target data extracted from the stream data;
A plurality of tables having a plurality of sets of one set of data composed of reference data corresponding to the keyword and reference destination table data and flag data corresponding to the reference data are stored, and the table according to the input of address data A memory that outputs a table corresponding to the address data; and
When a plurality of sets of data included in one table set are input from the memory, the target data and each reference data of the plurality of sets of data are compared, and if they match, the reference destination corresponding to the matched reference data A processing circuit for selecting table data and flag data and outputting the selected reference table data as address data in order to input a table corresponding to the selected reference table data from the memory; Programmable processor unit. The set of data further includes valid byte data, and the processing circuit corresponds to the matched reference data only when the reference data indicating that the valid byte data is valid matches the target data. 5. The programmable processor unit according to claim 4, wherein flag data and reference destination table data are selected. The programmable processor unit according to claim 4, wherein the flag data indicates whether or not the keyword is included in the stream data. 5. The programmable processor unit according to claim 4, wherein the memory stores a set of data including reference data, reference table data, and flag data corresponding to a plurality of keywords.

Images