JP2529466B2 - Separator - Google Patents

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a separator used in data processing using a computer.

2. Description of the Related Art Consider a process of finding a specified pattern from column-type data such as a byte string, and separating the column-type data immediately before or after the pattern.

Conventionally, such delimiter processing is performed by the programming language AW.
It is generally realized by a software program such as K (written by AV Eiho et al., Published by Toppan Co., Ltd.).

DISCLOSURE OF THE INVENTION Problems to be Solved by the Invention However, in the delimiter processing by a software program, there is a problem that the delimiter pattern has a high degree of freedom in designation, but the processing speed is slow.

In view of such a problem, the present invention scans data to be processed such as a byte string to search for a specified pattern, adds a specified data attribute value to a portion that matches the pattern, and outputs the processed data. For the purpose of clarifying the position and length of the pattern in the data, by accepting the pattern represented by the regular expression, and by making it possible to set multiple pairs of pattern information and data attribute value,
The purpose is not to lower the flexibility of pattern setting.

Means for Solving the Problems In order to solve the above-mentioned object, a data storage unit for storing processed data, a state transition table capable of storing a plurality of pairs of pattern information and a data attribute value, and the state transition table are used. The pattern was searched by the matching with the finite state automaton,
Further, at the time of data output, a collation unit that extracts the data attribute value from the state transition table and adds it to the output data and outputs it, a search address generation unit that generates a search address and stores the search end address, and a data output unit. And an output address generation unit that stores the output end address.

Function According to the present invention, the processed address stored in the data storage unit is scanned by the search address generation unit and the collation unit to search for the specified pattern. At this time, the search start address is a value following the address at which the previous search is completed. When the collation result is determined, the output address generation unit generates a data output address in order from the search start address until a match with the search end address stored in the search address generation unit is generated by a signal from the control unit. As a result, the data is again retrieved from the data storage unit and input to the collation unit. The collating unit adds the data attribute value extracted from the state transition table to the data and outputs the data. When the data output is completed, the pattern search is started again.

By continuing the above processing, the processed data is output unprocessed, and the data attribute value corresponding to the pattern is added to the pattern portion in the processed data and output. Therefore, by looking at the original data and the data attribute value added to it, the position and length of the pattern in the processed data can be clarified.

Also, since it is a collation by a finite state automaton, it is possible to accept a pattern represented by a regular expression. Furthermore, by making the state transition table an appropriate size, multiple pairs of pattern information and data attribute values can be received. Can be set individually. Therefore, the degree of freedom in pattern setting can be made approximately the same as in the case of the delimitation process by the software program.

Embodiment An embodiment of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram of a dividing device according to an embodiment of the present invention. In FIG. 1, 101 is a control unit for controlling the entire apparatus of the embodiment, 102 is a search address generation unit, 103 is an output address generation unit, 104 is an address selection unit for selecting either search address or output address, 105 Is a data storage unit for storing the data to be processed, 106 is a state transition table capable of storing a plurality of pairs of pattern information indicating a delimiter and data attribute value, and 107 is a finite state using the information in the state transition table 106. Data storage by collation by automaton
The pattern is searched from the data to be processed in 105, the corresponding data attribute value is taken out from the state transition table 106 at the time of data output, and is added to the output data to be output, and 108 is the data storage unit 105. A data input unit for storing, 109 is an address generation signal, 110 is an end signal indicating that the search address has reached the end of the data storage unit 105, 111 is a match signal indicating that the output address matches the search end address, 112 Is a selection signal, 113 is a plurality of control signals for controlling the operation of each unit, 114 is output data, 115 is a data attribute value corresponding to the output data, and 116 is input data.

Next, each of the constituent elements of the present invention in this embodiment will be described using block diagrams, and then the operation of the entire embodiment will be described.

A block diagram of the search address generator 102 is shown in FIG. In FIG. 2, 201 is an N-bit (N is a natural number) counter, 202 is an N-bit (N is a natural number) register, 2
Reference numeral 03 is an address generation signal, 204 is a search address, 205 is an end signal, and 206 is a control signal.

The operation of the search address generation unit shown in FIG. 2 will be described. When the address generation signal 203 is input, the value of the counter 201 is stored in the register 202 and output as the search address 204 at the rising edge. In addition, the counter 201 is activated by the falling edge of the address generation signal 203.
Update the value of. The end signal 205 indicating that the search address 204 has reached the end of the data storage unit 105 is generated by detecting the carry of the most significant digit of the counter 201. The value of the counter 201 is set to zero by the control signal 206.

A block diagram of the output address generator 103 is shown in FIG. In FIG. 3, 301 is an N-bit (N is a natural number) counter, 302 is an N-bit (N is a natural number) register, 3
Reference numeral 03 is an address generation signal, 304 is an output address, 305 is a comparison circuit, 306 is a match signal, 307 is a search end address, and 308 is a control signal.

The operation of the output address generator shown in FIG. 3 will be described. When the address generation signal 303 is input, the value of the counter 301 is stored in the register 302 and output as the output address 304 at the rising edge. Also, the counter 301 is generated by the falling edge of the address generation signal 303.
Update the value of. The comparison circuit 305 compares the output address 304 with the search end address 307, and if they match, a match signal 3
Outputs 06. The value of the counter 301 is set to zero by the control signal 308.

The data storage unit 105 is a randomly accessible storage device having addresses from 0 to 2 N.

A block diagram of the matching unit 107 is shown in FIG. In FIG. 4, 401 is a state register that holds a state, 402 is a state address generation circuit, 403 is a demultiplexer, 404 is data, 405 is output data, 406 is a state address, 407 is a state value from the state transition table, Reference numeral 408 is a data attribute value, 409 is status information, 410 is a selection signal, and 411 is a control signal.

Among the operations of the matching unit shown in FIG. 4, the operation during the matching process will be described first. The data 404 is input to the state address generation circuit 402 via the demultiplexer 403. The state address generation circuit 402 generates a state address 406 from the current state and the input data, and outputs it to the state transition table 106 according to the control signal 411. State transition table
The state value 407 obtained from 106 is stored in the state register 401 by the control signal 401, and is further output as the state information 409. Status value 4 stored in status register 401 at this time
07 is the new state after the state transition.

Next, the operation at the time of outputting data will be described. When the state value 407 obtained from the state transition table 106 is the end state, part of the state value 407 represents the data attribute value. Data attribute value 408 from status register 401 by selection signal 410
Is output. At the same time, the demultiplexer 403
Is set to output data 404 as output data 405. Therefore, the data 404 remains the unprocessed output data 4
In addition to being output as 05, the data attribute value 408 is also output. The control signal 411 can set the contents of the state register 401 to the initial state.

The state transition table 106 is composed of a randomly accessible storage device, and its size is determined depending on the type and number of patterns to be handled and the information amount of the data attribute value.

The control unit 101 includes a combination circuit and a sequential circuit, monitors the operation of each unit, and sends a control signal to each unit.

Next, the operation of the entire embodiment of the present invention will be described. In the present embodiment, there are four types of modes: an initial mode, a data input mode, and a data output mode. The details of the operation in these modes will be described below.

The initial mode is a state immediately after the control unit 101 initializes each unit via the control signal 113. At this time, the addresses output from the search address generation unit 102 and the output address generation unit 103 are both zero, which indicates the beginning of the data storage unit 105. Further, the selection signal 112 is set so that the address selection unit 104 selects a search address and the collation unit 107 further performs collation processing. Further, the state of the collating unit 107 is set to the initial state.

Next, the data input mode will be described. In this mode, the data input section is controlled by the control signal 113 from the control section 101.
It starts when 108 is started. The data input unit 108 sequentially stores the input data 116 from the beginning of the data recording unit 105 to immediately before the address indicated by the output address.

Next, the search mode will be described. The search mode is a mode for searching for a specified pattern from the data stored in the data storage unit 105.

The control unit 101 inputs the address generation signal 109 to the search address generation unit 102. Data for one processing unit is taken out from the data storage unit 105 according to the address from the search address generation unit 102. One processing unit is the amount of data processed by the matching unit 107 at one time. Next, in the matching unit 107,
The retrieved data is collated with the pattern information stored in the state transition table 106, and the collation unit 10 is collated according to the result.
The state 7 is transited, and further information regarding the new state after the state transition is transmitted to the control unit 101. The control unit 101 controls the connection of the search mode or the shift to the data output mode based on the input state information.

Continuation of the search mode is performed when the result of the state transition of the collation unit 107 is not the end state, that is, when the state of the collation unit 107 is in the midst of the collation process and the collation result is neither successful nor unsuccessful. Occurs in

When continuing the search mode, the control unit 101 inputs a new address generation signal 109 to the search address generation unit 102,
Next, the data to be collated is taken out from the data storage unit 105,
The following matching process is performed. The above operation is continued until the end state is reached. Note that during the search mode, the state of the output address generation unit 103 is maintained in the state at the end of the data output mode.

Next, the data output mode will be described. In the exploration mode described above, when the result of the state transition of the collation unit 107 reaches the end state, the mode shifts to this mode.

When shifting to this mode, the selection signal 112 from the control unit 101
Is set so that the address selecting unit 104 selects the output address and the collating unit 107 performs the data output process. Next, the control unit 101 outputs the address generation signal 109 to the address generation unit.
Enter in 103. The data for one processing unit is taken out from the data storage unit 105 according to the address from the output address generation unit 103, the data attribute value is added in the collation unit 107, and then output as the output data 114 and the data attribute value 115. The output start address is equal to the search start address in the previous search mode.

When the address from the output address generation unit 103 matches the search end address stored in the search address generation unit 102, the output address generation unit 103 sends a match signal 111 to the control unit 101. Upon receiving the coincidence signal 111, the control unit 101 ends the data output mode and shifts to the next processing.

When the address from the output address generation unit 103 does not match the search end address, the control unit 101 inputs a new address generation signal 109 to the output address generation unit 103 and continues the data output mode. Note that during the data output mode, the state of the search address generation unit 102 is maintained in the state at the end of the search mode.

When the data output mode ends, the search address generator
The control unit 101 checks whether the end signal 110 indicating that the search address has reached the end of the data storage unit 105 is output from 102. When the end signal 110 is output,
The control unit 101 shifts to the data input mode before shifting to the next exploration mode, and stores new processed data in the data storage unit.
Works like storing in 105.

EFFECTS OF THE INVENTION As described above, the present invention searches for a specified pattern from the processed data, adds the specified data attribute value to the found pattern, and outputs the pattern, thereby the position of the pattern in the processed data. Can be revealed. Also, because it is a collation by a finite state automaton,
It is possible to accept a pattern represented by a regular expression, and by setting the state transition table to an appropriate size, a plurality of pairs of pattern information and data attribute value can be set. Therefore, the degree of freedom in pattern setting can be made approximately the same as in the case of the division processing by the software program.

[Brief description of drawings]

FIG. 1 is a block connection diagram of a partitioning device according to an embodiment of the present invention, FIG. 2 is a detailed block connection diagram of a search address generation unit which is a main part of FIG. 1, and FIG. 3 is a block diagram of the output address generation unit. Detailed block connection diagram, FIG. 4 is a detailed block connection diagram of the matching unit. 101 ... Control unit, 102 ... Search address generation unit, 103 ... Output address generation unit, 104 ... Address selection unit, 105 ... Data storage unit, 106 ... State transition table, 107 ... Collation unit, 108 ... Data input unit, 201 ... Counter, 202 ... Register, 304 ... Output address, 305 ... Comparison circuit, 401 ... Status register, 402 ... Status address generation circuit, 403 ... Demultiplexer.

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Noboru Tamura 1006 Kadoma, Kadoma City, Osaka Prefecture Matsushita Electric Industrial Co., Ltd. Sangyo Co., Ltd. (72) Inventor Yoshihiro Hayakawa 1006 Kadoma, Kadoma City, Osaka Prefecture Matsushita Electric Industrial Co., Ltd. (56) Reference JP-A-63-73422 (JP, A)

Claims (1)

(57) [Claims] 1. A data storage unit for storing processed data,
A state transition table capable of storing a plurality of pairs of pattern information and data attribute values and a state transition table specified by performing collation by a finite state automaton using the state transition table while scanning the data stored in the data storage unit. The pattern is searched, and at the time of further data output, a data attribute value corresponding to the output data is taken out from the state transition table, a collating unit for adding and outputting the output data, and a search address are generated,
Further, a search address generation unit that stores the search end address and an address for data output are generated until they match the search end address stored by the search address generation unit, and an output address generation that further stores the output end address. And a separator.