JP2011008811A - Program, and data extraction method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To allow all of a required kind of data to be quickly obtained from even large quantities of data with respect to a technique for extracting data meeting a designated extraction condition from available data.SOLUTION: A plurality of extraction conditions different by target data can be input, and in response to input of one or more extraction conditions, data are extracted under each extraction condition and extracted data are output to output destinations corresponding to extraction condition which the data meets. Accordingly, a user can obtain a plurality of extraction results simultaneously by defining and input of a plurality of extraction conditions. Thus, all of required extraction results can be quickly obtained, and as a result, high work efficiency can be easily achieved.

Description

  The present invention relates to a technique for extracting data satisfying a specified extraction condition from data that can be acquired.

  A data extraction apparatus that can extract arbitrary data from obtainable data is currently widely used in various applications. It is used as a search engine for searching information published on the Internet. The user can quickly obtain desired data from a large amount of data by using the data extraction device.

  The data extraction device extracts data in a predetermined unit. The unit is, for example, a file or a record. Documents and Web pages on the Internet correspond to files. Customer usage record POS (Point Of Sales) data, HHT (Hand Held Terminal) data, and the like are usually managed in record units.

FIG. 1 is a diagram for explaining a conventional data extraction method. Here, the data extraction method will be specifically described with reference to FIG.
The conventional data extraction method shown in FIG. 1 is for example performed in a credit card company. The notation “JOURNAL” represents a journal file in which fact data is stored in units of records. “MASTER” represents a master file in which data of customers who are credit card owners are stored in units of records. Accordingly, the data extraction method shown in FIG. 1 uses SQL (Structured Query Language) to connect (JOIN) desired ones from a plurality of existing journal files and master files, and from the result of the connection. The example in the case of extracting a desired record is represented.

  Each condition of the journal file and the master file to be linked is described in the WHERE clause in the FROM clause. Based on the conditions described there, the current master file is selected, and the journal file is selected for 2004. The FROM phrase in the FROM phrase describes that the correspondence between records between files is specified by a credit card number. The data items stored in the record extracted from the concatenation result are described in the SERECT phrase. The items described there are the customer's nomination (V.NAME), its age (V.AGE), the number of uses (V.SALES_NUM), and the sales amount (V.SALES). The condition of the record extracted from the concatenation result is described in the WHERE clause. The condition described there is that the card type is a cold card. For this reason, a record of a customer who used in 2004 and still has a gold card is extracted as a search result.

  In order to make the records extracted from the concatenation result different, the extraction condition described in the WHERE clause may be changed. If a record of a customer having a silver card is to be extracted, for example, as shown in FIG. 2, the description of “GOLD” may be changed to “SILVER”. Thereby, a record of a customer who used in 2004 and still has a silver card is extracted as a search result.

  As described above, in the conventional data extraction method, an extraction condition for obtaining desired data is determined, and a search is performed for each extraction condition. For this reason, as the number of objectives for extracting data, that is, the number of extraction conditions used for the search increases, the time required to obtain all extraction results becomes longer, and the efficient work cannot be performed. It was.

  Currently, the types and amounts of information handled by digital data are increasing greatly. Therefore, it is expected that it will become very difficult to cope with the conventional data extraction method in the future. For this reason, it is important to make it possible to obtain all necessary types of data more quickly from a huge amount of data.

JP 2002-222194 A JP-A-2005-70911 JP-A-6-319906

An object of the present invention is to provide a technique that enables all necessary types of data to be obtained more quickly from a vast amount of data.
Both the programs of the first and second aspects of the present invention are premised on causing a computer to execute a data extraction apparatus that can extract data satisfying a specified extraction condition from among obtainable data. Each implements the following functions.

  The program according to the first aspect includes a function for acquiring data, a function for inputting extraction conditions, and a function for extracting data for each extraction condition using one or more input extraction conditions. And a function of outputting the data extracted for each extraction condition to different output destinations by the extracting function.

  The program according to the second aspect divides the conditional expression constituting the extraction condition input by the function of acquiring data, the function of inputting the extraction condition, and the function of inputting into a plurality of partial conditional expressions. In the data acquired by the function to acquire by converting the extraction condition into a format expressed by the combination of the obtained partial conditional expressions and confirming whether or not the partial conditional expression satisfies the partial conditional expression unit And a function of extracting data satisfying the extraction condition.

  The data extraction method of the present invention is premised on being applied to extract data satisfying a specified extraction condition from among obtainable data, and a plurality of extraction conditions with different target data can be input. When one or more extraction conditions are input, data is extracted for each extraction condition, and data obtained by the extraction is output to an output destination corresponding to the extraction condition satisfied by the data.

  In the present invention, a plurality of extraction conditions with different target data can be input, and when one or more extraction conditions are input, data is extracted for each extraction condition, and the data obtained thereby is The data is output to output destinations corresponding to the extraction conditions satisfied by the data. For this reason, the user can obtain a plurality of extraction results at a time by defining and inputting a plurality of extraction conditions. Thereby, all necessary extraction results can be obtained more quickly. As a result, high work efficiency can be easily realized.

  In the present invention, the input extraction condition is divided into a plurality of partial conditional expressions and converted into a form expressed by a combination of partial conditional expressions obtained by the division, and the partial conditional expression unit By checking whether or not the partial conditional expression is satisfied, data satisfying the extraction condition is extracted from the data. By converting the extraction condition to a format that is expressed by a combination of partial conditional expressions, even if the same partial conditional expression exists in different conditional expressions, check whether the data satisfies the partial conditional expression for each conditional expression The need to do so can be avoided. For this reason, data extraction can be performed with a smaller load.

It is a figure explaining the conventional data extraction method. It is a figure explaining the difference in the extraction conditions for extracting different types of data with the conventional data extraction method. It is a figure explaining the yesterday structure of the data extraction device by this Embodiment. It is a figure explaining the data extraction which the data extracting device 100 by this Embodiment can perform. It is a figure which shows an example of the hardware constitutions of the computer which can implement | achieve the data totaling apparatus by this Embodiment. It is a figure explaining the structural example of XML data. It is a figure explaining the structural example of CSV data. It is a figure explaining the example of the content of an extraction condition group. It is a figure explaining the tag DFA example. It is a figure explaining the hierarchy collation NFA example. It is a figure explaining the CSV analysis DFA example. It is a figure explaining the example of keyword DFA. It is a figure explaining the example of a logical table. It is a figure explaining the management method of an output buffer. It is a flowchart of the process which the extraction condition input part 110 performs. It is a flowchart of the process which the data input structure search part 120 performs. It is a flowchart of the process which the extraction condition determination part 130 performs. It is a flowchart of the process which the data determination part 140 performs. It is a figure explaining the example of application of the data extracting device by this Embodiment (the 1). It is a figure explaining the application example of the data extraction apparatus by this Embodiment (the 2). It is a figure explaining the application example of the data extraction apparatus by this Embodiment (the 3). It is a figure explaining the application example of the data extraction apparatus by this Embodiment (the 4). It is a figure explaining the application example of the data extraction apparatus by this Embodiment (the 5). It is a figure explaining the example of application of the data extraction apparatus by this Embodiment (the 6).

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 3 is a diagram for explaining the functional configuration of the data extraction apparatus according to this embodiment.
The data extraction apparatus 100 is implemented as text data input from the input device 210 as data 211, and the data 211 is distributed according to a specified extraction condition group 220 and output. For this purpose, an extraction condition input unit 110, a data input structure search unit 120, an extraction condition determination unit 130, a data determination unit 140, an output buffer 150 for external output, and a data output unit 160 are provided. Here, for convenience, it is assumed that only XML (eXtensible Markup Language) data as shown in FIG. 6 and CSV (Comma Separated Values) data as shown in FIG. Both of these data are text data.

  The extraction condition group 220 input by the extraction condition input unit 110 has contents as shown in FIG. 8, for example. In FIG. 8, the extraction condition and the output condition are respectively shown in (1) to (3). All the extraction conditions shown separately are for the user to extract the desired data 211. The output condition shown together with the extraction condition specifies the output destination of the data 211 extracted by the extraction condition and its file name. Thereby, the extraction condition group 220 designates the extraction condition to be satisfied by the data 211 and the output destination file name for each desired data 211. The reason why the output destination of the data 211 can be arbitrarily designated is to enable the data 211 to be used in a desired form more quickly. Hereinafter, the extraction condition described in (1) is referred to as “extraction condition 1”. The same applies to other cases.

FIG. 4 is a diagram for explaining data extraction that can be performed by the data extraction device 100 according to the present embodiment. Here, the data extraction will be specifically described with reference to FIG.
The extraction condition group 220 shown in FIG. 8 assumes XML data as the data 211. FIG. 4 shows an extraction condition group 220 assuming CSV data. “Query” corresponds to the extraction condition, and “OutFile” corresponds to the output condition. “$ X” written as “Query” (extraction condition) represents an item name “X”, and “$ _” represents an arbitrary item name. Accordingly, for example, “$ X ==“ X1 ”OR $ X ==“ Xa ”” expressed in Query 1 indicates that data 211 whose data of item name “X” is X1 or Xa is an extraction target. Show. The query whose notation is “$ _ ==“ Xa ”” indicates that data 211 in which Xa exists as data of an arbitrary item is an extraction target. The data 211 may be XML data or CSV data, which may be input together as a file, but may be input sequentially one by one. When the data is input one by one, the XML data is as shown in FIG. 6, and the CSV data is like a line having “000001” to “000007” at the head in FIG. Here, for convenience, a collection of such data is called a record. A character string described between two “′” is called a “keyword”. The keyword corresponds to a character string described between two """in the extraction condition group 220 shown in FIG.

  In the present embodiment, the data 211 that satisfies any of the extraction conditions specified in the extraction condition group 220 is extracted using the character string matching method, and the output specified in the output condition associated with the extraction condition that satisfies Output to the file with the destination file name. As a result, the data 211 satisfying Query 1 is the file 231 with the file name “result1.csv”, the data 211 satisfying Query 2 is the file 232 with the file name “result2.csv”, and the data 211 satisfying Query 3 is the file name “result3.csv”. csv "files 233, respectively. The correspondence between the input data 211 and the data 211 output to any one of the files 231 to 231 is indicated by notations (1) to (6) in the figure.

  Since each extraction condition is considered independently, all the extraction conditions can be arbitrarily defined. Therefore, one or more extraction conditions can be defined for each type of data 211 such as XML data and CSV data, and one or more extraction conditions can be defined for each structure. Yes. Therefore, no matter how the schema differs between the target data 211, the influence of the difference can be reliably avoided.

  As described above, the extraction conditions may not be exclusive. As a result, Query 1 and Query 2 have contents for extracting data 211 that satisfies the conditional expression (logical expression) “$ X == 'Xa'”, respectively. Similarly, in Query 2 and Query 3, the contents satisfying the conditional expression “$ X == 'Xb'” are extracted. As a result, data 211 expressing (4) is output to both the files 231 and 232, and data 211 expressing (5) is output to both the files 232 and 233.

  As described above, when a plurality of extraction conditions are designated by the extraction condition group 220, the data 211 satisfying the extraction conditions is sorted and output to a designated output destination. For this reason, the user can obtain a plurality of extraction results at a time only by defining a plurality of extraction conditions and output conditions as the extraction condition group 220. Thereby, all necessary extraction results can be obtained more quickly. As a result, high work efficiency can be easily realized.

  As described above, the present embodiment employs the character string matching method. The character string collation method is such that the character string specified in the extraction condition and the target data 211 are collated sequentially from the beginning of the data 211 toward the rear, so that the character string is included in the data 211. It is to check whether or not it exists. In the character string collation method, it is possible to confirm which of the extraction conditions defined in the extraction condition group 220 satisfies the data 211 by performing only one scan from the top to the back. Therefore, it is possible to always extract data 211 to be extracted quickly regardless of the number of defined extraction conditions. Examples of the reference include Patent Documents 1 and 2.

Returning to the description of FIG.
The extraction condition input unit 110 inputs the extraction condition group 220 as described above, analyzes the extraction condition for each extraction condition, and generates a corresponding automaton. Accordingly, if the extraction condition is for XML data, a tag DFA (Deterministic Finite state Automaton) 170, a hierarchical matching NFA (Non-deterministic Finite state Automaton) 171 and a keyword DFA 180 are generated. If the extraction condition is for CSV data, a CSV analysis DFA 172 and a keyword DFA 180 are generated. Similar to the keyword DFA 172, the logical table 190 is generated regardless of the type of data 211 assumed by the extraction condition.

  Creation of the extraction condition group 220 is basically performed by data input by the user. When the extraction condition group 220 is created by a terminal device connected to the data extraction apparatus 100 according to the present embodiment, for example, the user displays a screen for creating the extraction condition group 220 and the extraction condition of desired contents is displayed on the screen. The group 220 is input. When data extraction is instructed after the input, the created extraction condition group 220 is output to the data extraction apparatus 100.

  As the logical table 190, when the extraction condition group 220 has the content shown in FIG. 8, the extraction condition input unit 110 generates the logical table 190 as shown in FIG. As shown in FIG. 13, the logical table 190 includes an A logical table 190a and a Z logical table 190b.

The A logical table 190a decomposes the conditional expression (logical expression) constituting the extraction condition with a relational operator (corresponding to “=” and “<” in FIG. 8), and subdivides it according to the logic expressed by the conditional expression. (In FIG. 8, the conditional expression “/ root / Company / code <
“99” is decomposed into “/ root / Company / code” and “<99”), and each subdivided conditional expression (partial conditional expression) is assigned a unique logical number. The Z logic table 190b has a configuration in which a conditional expression or an extraction condition is expressed by a partial conditional expression or a combination of logical numbers attached to the conditional expression, and a unique logical number is assigned to each expressed combination. The logical number to be combined may be any of the A logical table 190a and the Z logical table 190b. By expressing a conditional expression or an extraction condition using the logical number, a record (row) to be referred to in the A logical table 190a or the Z logical table 190b can be specified. Although not specifically shown, the Z logical table 190b can store a conditional expression expressed by the combination of each logical number or a code indicating whether or not the extraction condition is satisfied. It has become. Hereinafter, in order to distinguish the logical numbers assigned in the tables 190a and 190b, the logical numbers in the A logical table 190a are indicated with “A”, and the logicals in the Z logical table 190b are indicated with “Z” at the head. .

  The combination to which the logical number Z1 is assigned in the Z logical table 190b is “A1 × A2”. In the combination “A1 × A2”, the data 211 in which the partial conditional expression (/ root / origin) of the logical number A1 is satisfied and the partial conditional expression (“atcg”) of the logical number A2 is satisfied is to be extracted. It is a logical expression of the form representing Thereby, “x” in the combination (logical expression) “A1 × A2” is a logical operator indicating that logical product of the partial conditional expressions of the logical numbers A1 and A2 is performed. The logical expression represents the contents of the extraction condition 1. Similarly, logical expressions with logical numbers Z4 and Z5 represent the contents of extraction conditions 3 and 2, respectively. The extraction condition 2 is Z5 = Z2 × Z3. Here, in the table of 190b, Z2 = A3 × A4 corresponds to A3 = / root / Company / code, A4 = <99.

  In addition, Z3 = A1 × A5 corresponds to A1 = / root / origin and A5 = “gtac”. Therefore, the extraction condition 2 corresponds to the A logical numbers A3, A4, A1, and A5 through the Z logical number Z5, and the logical product (AND) of the extraction condition 2 shown in FIG. 8 is the logical table shown in FIG. And the link state between the elements. The extraction condition 3 in FIG. 8 is indicated by the extraction condition 3, the logical table of Z logical number 4, A logical number A1, A6 and the link between the elements. That is, the extraction condition 3 corresponds to the A logical number as Z4 = A1 × A6 (A1 = / root / origin, A6 = “aacg”). That is, it is possible to determine data for each extraction condition using a logical table formed by each extraction condition with such a logical number.

  The search result determination information 195 illustrated in FIG. 13 includes, for each extraction condition, an output buffer 150 in which a logical number given to a combination of logical numbers expressing the extraction condition and data 211 satisfying the extraction condition are to be stored. The numbers shown (indicated as “output buffer No.” in the figure) and file descriptors (corresponding output conditions) are collected. As a result, the data 211 satisfying any one of the extraction conditions is output to the output buffer 150 to be output with reference to the search result determination information 195 and then to the file to be output.

  The automaton (tag DFA 170, hierarchical collation NFA 171, keyword DFA 180, CSV analysis DFA 172) is a state transition table for collating the character string in the search condition with the data 211. The states are represented by connecting arrows indicating the direction of transition. The head is set as an initial state, and the state is sequentially shifted from the initial state according to the character string in the data 211. The transition state includes one or more acceptance states corresponding to the character located at the end of the character string in the search condition. As a result, the automaton is generated so as to transition to any of the accepting states if there is a character string to be detected in the data 211. When transitioning to an acceptance state, hit information corresponding to the acceptance state is output. The hit information is peculiar to the transitioned acceptance state, and is generated when the automaton is generated.

  The tag DFA 170 is for detecting a search path up to an element having a character string (element content) to be matched with a keyword. When the extraction condition group 220 has the contents shown in FIG. 8, the extraction condition input unit 110 finally generates a tag DFA 170 as shown in FIG. In the extraction condition group 220 shown in FIG. 8, since there are “/ root / origin” and “/ root / Company / code” as search paths, the character strings “root”, “origin”, “Company”, which are tag names, respectively. And “code” can be detected. The transition to the acceptance state corresponding to any of the characters “t”, “n”, “y”, and “e” located at the end of those character strings indicates that the character string corresponding to the character has been detected. One of the pieces of hit information 170a-d shown is output.

  The hierarchical collation NFA 171 is for managing the target search path at present. When the extraction condition group 220 has the contents shown in FIG. 8, the extraction condition input unit 110 finally generates a hierarchical collation NFA 171 as shown in FIG. As shown in FIG. 10, the NFA 171 is generated so that state transition is performed in units of tag names described in any of the search paths. Therefore, the state transition is generated by the start tag and the end tag. Here, a state where “4” and “2” are written corresponds to an accepted state.

  The transition to the acceptance state with “4” means that the search path “/ root / Company / code” has been detected. As a result, in the node specified by the search path, hit information 171a for collating whether or not the value is less than 99, that is, whether or not the partial conditional expression (logic) of the logical number A4 is satisfied is output. Is done. The hit information 171a indicates a logical number (in this case, A4) indicating a partial conditional expression to be collated, hierarchical information indicating the depth of the search path hierarchy, and the content whose relationship should be confirmed by the partial conditional expression. It contains comparison information (here <99). Similarly, the transition to the accepting state with “2” means that the search path “/ root / origin” has been detected. Therefore, the node specified by the search path, that is, the tag name “origin” In the tag, hit information 171b-d for collating whether the character string matches any of “atcg”, “gtac”, or “aacg” is output. The reason why the comparison information is not indicated in the hit information 171b-d is that the partial conditional expression corresponding to the logical number written in them is collated by the keyword DFA180.

  The state transition in the hierarchical collation NFA 171 is performed using a tag DFA 170 shown in FIG. For example, when the character string “root” which is the tag name is detected by the tag DFA 170, that is, when the hit information 170a is output by the tag DFA 170, the NFA 171 transitions from the initial state in which “0” is described to the state in which “1” is described. Next, when the character string “origin” is detected by the tag DFA 170, the NFA 171 transitions from a state where “1” is written to a state where “2” is written. At this time, when the character string “Company” is detected by the tag DFA 170, the NFA 171 transitions from a state where “1” is written to a state where “3” is written. If any of these character strings cannot be detected by the tag DFA 170, the NFA 171 transitions from a state in which “1” is written to an initial state in which “0” is written. By making such a transition, the presence / absence of movement of the hierarchy along the search path is grasped using the hierarchy verification NFA 171 and the target search path is managed.

  The CSV analysis DFA 172 is for detecting a search path up to an element having a character string (element content) to be matched with a keyword. In the CSV data (FIG. 7) in which the element exists between two double quotations, the extraction condition input unit 110 generates a CSV analysis DFA 172 as shown in FIG. “0x” shown in FIG. 11 indicates that the following symbol is expressed in hexadecimal.

  The keyword DFA 180 is for detecting from the data 211 a character string that matches the keyword specified by the extraction condition. When the extraction condition group 220 has the content shown in FIG. 8, the extraction condition input unit 110 finally generates a keyword DFA 180 as shown in FIG. When a transition is made to an acceptance state corresponding to the last character of any of the registered keywords, that is, when any of the character strings “aacg”, “acgt”, and “gtac” can be detected, the detected character One of the hit information 180a to 180c is output according to the column.

  The data input structure search unit 120 continuously inputs data 211 by a predetermined amount from the input device 210, and determines an automaton to be used for matching according to the type of the data 211. Thus, if the data 211 is XML data, the search path described in any of the extraction conditions is detected using the tag DFA 170 and the hierarchical collation NFA 171. If the data 211 is CSV data, the CSV analysis DFA 172 is used to detect the item name described in any of the extraction conditions. When the search path or item name is detected, the data specified by the search path or the data position information indicating the position where the cell of the item name starts and the node / cell information indicating the detected character string are extracted. The determination unit 130 is notified. Such information is generated, for example, as hit information or includes it. Notification of such information is performed every time a search path or item name is detected until the end of the data 211 is detected. The end detection corresponds to detection of an end tag paired with a root tag in XML data, and corresponds to detection of a predetermined number of cells in CSV data. Detection of a search path or item name by the data input structure search unit 120 corresponds to confirmation that the partial conditional expression stored in the A logic table 190a is satisfied.

  The extraction condition determination unit 130 performs collation using the keyword DFA 180 based on the data position indicated by the data position information notified from the data input structure search unit 120. As a result of the comparison, it is confirmed that there is a character string that matches any keyword from the data position or a value that satisfies the relationship indicated by the relational operator (value less than 99 in the extraction condition group 220 shown in FIG. 8). Then, a code indicating this (hereinafter referred to as “true code” and a different code as “false code”) is stored at the corresponding logical number in the Z logic table 190b. If the end of the data 211 is detected before the confirmation, the data input structure search unit 120 is notified of the data position information indicating the position of the end. Thereby, the structure search unit 120 notifies the data determination unit 140 that the scanning has been completed up to the end of the data 211 regardless of whether or not the end of the data 211 has been detected.

  The extraction condition determination unit 130 performs collation using the keyword DFA 180 every time information is notified from the structure search unit 120 until the notification is performed or the structure search unit 120 detects the end. As a result, when the data 211 satisfies the extraction condition 2, the true code is sequentially stored as the codes of the logical numbers Z2 and Z3, and finally the true code is stored as the code of the logical number Z5. Become. In this way, since the true code is stored only at the location of the logical number that satisfies the logical expression of the target data 211, the extraction condition that the data 211 satisfies can be confirmed by referring to the Z logical table 190b. It has become.

  In this way, in this embodiment, the conditional expressions constituting the extraction condition are subdivided according to the logic expressed by the conditional expressions, and collation is performed in units of partial conditional expressions (subdivided logic) obtained by the subdivision. ing. Thereby, detection of matching character strings or search paths, confirmation of relations represented by relational operators, and identification of locations where such a thing should be performed are performed individually. By doing so, it becomes possible to respond more flexibly, and even if the information such as the type and structure of the data 211 is insufficient, the user can obtain the content that the desired data 211 satisfies from the obtained information. This makes it easier to define the extraction conditions. For this reason, high convenience for the user is realized.

  Partial conditional expressions (subdivision logic) may exist separately under the same or other extraction conditions. In the example illustrated in FIG. 8, the partial conditional expression “/ root / origin” is described in any of the extraction conditions 1 to 3. However, such a plurality of the same descriptions can be left as one partial conditional expression by subdividing the conditional expression. As a result, regardless of the number and contents of extraction conditions, the partial conditional expressions to be confirmed whether or not they are satisfied can be suppressed to the minimum necessary. A conditional expression or extraction condition is expressed by a combination of a plurality of partial conditional expressions. For this reason, whether or not they are established can be performed more quickly.

  The data determination unit 140 refers to the Z logic table 190b and confirms the extraction condition that the data 211 satisfies. If it is determined by the confirmation that any one of the extraction conditions is satisfied, the data 211 is output and stored in the output buffer 150 to be output with reference to the search result determination information 195 (FIG. 13).

FIG. 14 is a diagram for explaining an output buffer management method.
Output of the data 211 to the corresponding output buffer 150 is managed by output buffer information 151 and buffer information 152. The output buffer information 151 includes acquisition buffer number information indicating the number of output buffers 150 secured by the extraction condition group 220 and pointer information for accessing the buffer information 152. The buffer information 152 includes the number of records indicated by the acquisition buffer number information, and each record includes a plurality of pieces of information related to the corresponding output buffer 150 (here, one of the output buffers 150a to 150c). The individual buffer information 153 (in this case, one of the individual buffer information 153a to 153c) is stored. The output buffer information 151 and the area for storing the buffer information 152 are secured on the storage device 1401 mounted on or connected to the data extraction device 100 together with the output buffer 150. The tag DFA 170, the hierarchical collation NFA 171, the CSV analysis DFA 172, the keyword DFA 180, and the logical table 190 are also stored in the storage device 1401, for example.

  The individual buffer information 153 includes pointer information for accessing the corresponding output buffer 150, a total buffer size indicating the total size in which the data 211 can be stored, and a remaining size in which the data 211 can be stored. A remaining buffer size, and an output buffer size representing the size of the secured output buffer 150 itself. The size relationship between the numbers assigned to each record is the same as that of the extraction condition numbers. That is, the record with record number 0 corresponds to the extraction condition 1. Thereby, the record corresponding to the extraction condition satisfied by the data 211 can be specified.

  As described above, when the data determination unit 140 refers to the Z logic table 190b and confirms that there is an extraction condition that the data 211 satisfies, the data determination unit 140 refers to the search result determination information 195 to determine the extraction condition. Confirmation is made and the output buffer information 151 and the buffer information 152 are referred to. Thereby, the record corresponding to the confirmed extraction condition is taken out from the buffer information 152, and the data 211 is output to the output buffer 150 designated by the individual buffer information 153 stored in the record. The remaining buffer size is updated according to the size of the data 211 to be output.

  The data output unit 160 monitors, for example, the remaining buffer size of each output buffer 150, and determines the search result when the size is equal to or smaller than a predetermined value or when there is no data 211 to be input and processed from the input device 210. With reference to the information 195, the data 211 stored in the output buffer 150 is output to the corresponding file. Thereby, the data 211 extracted so far is stored in the file having the output destination file name designated by the output condition. Here, the three files 231 to 233 are all stored on the same output device 230.

  FIG. 5 is a diagram illustrating an example of a hardware configuration of a computer that can implement the data extraction apparatus 100. Although the extraction device 100 may be realized by a plurality of computers (data processing devices), it will be described here on the assumption that it is realized by a single computer having the configuration shown in FIG.

  The computer shown in FIG. 5 includes a CPU 51, a memory 52, an input device 53, an output device 54, an external storage device 55, a medium drive device 56, and a network connection device 57, which are connected to each other via a bus 58. It has become. The configuration shown in the figure is an example, and the present invention is not limited to this.

  The memory 52 is a memory such as a RAM that temporarily stores data. A program or data stored in the portable recording medium MD accessed by the external storage device 55 or the medium driving device 56 is temporarily stored. The CPU 51 performs overall control by reading the program into the memory 52 and executing it. The program may be acquired by the network connection device 57 via the network.

  The input device 53 is connected to or has input devices such as a keyboard and a mouse, for example. The user's operation on such an input device is detected, and the detection result is notified to the CPU 51.

The output device 54 is connected to or has a display, for example. The data sent under the control of the CPU 51 is output on the display.
The network connection device 57 is for communicating with other devices via a network such as an intranet or the Internet. The external storage device 55 is a hard disk device, for example. Mainly used for storing various data and programs.

  The storage medium driving device 56 accesses a portable recording medium MD such as a flexible disk, an optical disk (including CD-ROM, CD-R, and DVD), or a magneto-optical disk.

  In the configuration shown in FIG. 5, the output device 230 shown in FIG. 3 corresponds to the external storage device 55, the medium drive device 56 loaded with the recording medium MD, or an external device accessible by the network connection device 57. The input device 210 corresponds to a medium driving device 56 in which the recording medium MD is mounted or an external device accessible by the network connection device 57. The extraction condition group 220 can be input by the input device 53, the medium driving device 56 to which the recording medium MD is mounted, or the network connection device 57. A storage device 1401 illustrated in FIG. 14 corresponds to at least one of the external storage device 55 and the memory 52, for example.

  The search condition input unit 110 is realized by the units 51 to 53 and 55 to 58 excluding the output device 54, for example. Both the data input structure search unit 120 and the data output unit 160 are realized by the units 51, 52, and 55 to 57, excluding the input device 53 and the output device 54, for example. Both the extraction condition determination unit 130 and the data determination unit 140 are realized by the respective units 51, 52, 55, 56, and 58 excluding the input device 53, the output device 54, and the network connection device 57, for example.

  Next, the operation of each of the above-described units 110, 120, 130, and 140 will be described in detail with reference to the flowcharts of the processes shown in FIGS. All of these processes are realized by, for example, the CPU 51 reading out the program stored in the portable storage medium MD mounted on the external storage device 55 or the medium driving device 56 to the memory 52 and executing it.

  FIG. 15 is a flowchart of processing executed by the extraction condition input unit 110. First, the process will be described in detail with reference to FIG. The process is started, for example, when the user instructs input of the extraction condition group 220 via the input device 53 or the network. In that case, the extraction condition group 220 is input via the input device 53 or the network connection device 57.

  First, in step 11, the extraction condition group 220 is input and stored in the memory 52, for example. In the subsequent step 12, one extraction condition is selected from the stored extraction condition group 220 and read out, and analyzed to identify the corresponding automaton type. In the next step 13, the specified type of automaton is generated or updated. By the generation or update, the character string described in the extraction condition is registered in the tag DFA 170, the hierarchical collation NFA 171 or the keyword DFA 180 as necessary.

  In step 14 following step 13, it is determined whether there are other extraction conditions not selected in the extraction condition group 220. If such an extraction condition remains, the determination is yes and the process returns to step 12 to select another selection condition. Otherwise, the determination is no, and in step 15, the search result determination information 195 (FIG. 13), the output buffer information 151, and the buffer information 152 are generated together with the generation of the logical table 190, and the number of extraction conditions is set. After securing the corresponding output buffer 150 (FIG. 14), the series of processing ends. In this way, by inputting the extraction condition group 220, preparation for outputting the data 211 to an output destination to be output is performed together with generation of a necessary automaton.

  FIG. 16 is a flowchart of processing executed by the data input structure search unit 120. Next, the processing will be described in detail with reference to FIG. The process is executed while an instruction to fetch the data 211 from the input device 210 is given, for example.

  First, in step 21, it is determined whether there is data 211 to be input from the input device 210. If there is no such data 211, the determination is no and the determination is performed again. Thereby, it waits for the data 211 to be generated. On the other hand, when that is not right, determination will be YES and it will transfer to step 22.

  In step 22, a predetermined amount of data 211 is input from the input device 210. In the subsequent step 23, one of the input data 211 is selected, and a character string that matches any of the character strings registered in it is searched using the automaton determined by the extraction condition input unit 110.

  The search is performed in units of one character. When the search is completed, the process proceeds to step 24 to determine whether or not the target character string (search path, item name, etc.) has been detected. If such a character string cannot be detected, the determination is no and the process moves to step 27. Otherwise, the determination is yes and the process moves to step 25.

  In step 25, the data position information and the like are notified to the extraction condition determination unit 130. In response to the notification, the extraction condition determination unit 13 performs collation using the keyword DFA 180. When the end of the data 211 is detected by the collation, the data position information is notified. Therefore, in the next step 26, it is determined whether or not there is a notification. If there is such notification, the determination is yes and the process moves to step 28. Otherwise, the determination is no and the process returns to step 23 to continue the search.

  In step 27 where the determination in step 24 is NO and the process proceeds, it is determined whether or not the end of the data 211 has been detected by the search. If the end is detected, the determination is yes and the process moves to step 28. Otherwise, the determination is no and the process returns to step 23 to continue the search.

  In step 28, the data determination unit 140 is notified that the end of the data 211 has been detected. In the following step 29, it is determined whether or not there is unselected data 211 in the input data 211. If unselected data 211 exists, the determination is yes, the process returns to step 23, the unselected data 211 is selected, and the search is started. Otherwise, the determination is no and the process returns to step 21 above. Thereby, it is confirmed whether or not there is data 211 to be input to the input device 210.

FIG. 17 is a flowchart of processing executed by the extraction condition determination unit 130. Next, the processing will be described in detail with reference to FIG.
First, in step 41, it waits for notification of the end of record. When the notification is received, the determination is no, the process proceeds to step 42, and collation is performed using the notified data position information and the keyword DFA 180. In the next step 43, it is determined whether or not a character string that matches any of the keywords registered in the keyword DFA 180 has been detected from the data 211. If such a character string can be detected, the determination is YES, a true code is set at the position of the corresponding logical number in the logical table 190 (Z logical table 190b) in step 44, and the process returns to step 41 to wait for notification. Transition to the state. Otherwise, the determination is no and the process moves to step 45.

  In step 45, it is determined whether or not the end of the data 211 has been detected. If the end is detected by collation, the determination is yes, and the data position information is notified to the data input structure search unit 120 in step 46 in order to notify that, and the process returns to step 41. Otherwise, the determination is no and the process returns to step 42 and the collation is continued.

  As described above, necessary information is exchanged at any time between the data input structure search unit 120 and the extraction condition determination unit 130, and processing is advanced according to the information. Thereby, for each data 211, an extraction condition for satisfying it is confirmed, and processing according to the confirmation result is performed.

FIG. 18 is a flowchart of processing executed by the data determination unit 140. Finally, the processing will be described in detail with reference to FIG.
First, in step 51, it waits for the end of the data 211 to be notified from the data input structure search unit 120. When the notification is received, the determination is no, the process proceeds to step 52, and the logical table 190 is referenced to determine the extraction condition that the currently targeted data 211 satisfies. Thereafter, the process proceeds to step 53.

  In step 53, it is determined whether there is an extraction condition that the data 211 satisfies. If such an extraction condition exists, the determination is YES, the process proceeds to step 54, and data is obtained with reference to the search result determination information 195 (FIG. 13), the output buffer information 151, and the buffer information 152 (FIG. 14). 211 is output to the output buffer 150 to be output, and the corresponding individual buffer information 153 is updated. Thereby, it shifts to a notification waiting state. On the other hand, if not, the determination is no and the process returns to step 51.

  19 to 24 are diagrams for explaining application examples of the data extraction device. Hereinafter, the applicable usage will be described in detail with reference to FIGS. In FIG. 19 to FIG. 24, the data extraction device is expressed as “extractor”.

  FIG. 19 shows an example in which a plurality of data extraction devices 100 are used in multiple stages. The data extraction apparatus 100 that receives the data 1903 distributes the data 1903 to the two connectors 1910. One of the two concatenators 1910 concatenates the data of the master file 1901 with the data 1903 and outputs it to another data extraction device 100, which distributes the concatenation result to the two totalizers 1920. . The two tabulators 1920 output the tabulation results to different data extraction devices 100, and the data extraction device 100 that inputs the tabulation results sorts the data into three files and outputs them. The same applies to the other side of the two couplers 1910.

  FIG. 20 shows an example in which the data extraction device 100 is used for sorting input data. The input data is data of each record stored in the journal file 2000. The data extraction device 100 is used to distribute and output data satisfying the extraction condition to any of the journal files 2001 to 3. The reason for such distribution is, for example, to cope with different connection conditions with the masters X to Z. If distributed in this way, it is possible to process data in parallel in three systems, so that the processing speed can be increased.

  FIG. 21 shows an example in which the data extraction apparatus 100 is used for sorting the data of the connection result. The concatenation result is obtained by concatenating master and journal data. The data extraction device 100 is used to output data satisfying any one of the extraction conditions 1 to 3 to any of the files 2101 to 3 according to the extraction condition.

  FIG. 22 shows an example in which the data extraction device 100 is used for sorting the data of the total results. The aggregation result is obtained by performing an aggregation operation on the result of concatenating master and journal data. The data extraction apparatus 100 is used to output the data of the aggregation result that satisfies any of the extraction conditions 1 to 3 to any of the files 2201 to 3 according to the extraction condition.

  FIG. 23 shows an example in which the data extraction apparatus 100 is used for providing a clipping service implemented in a newspaper company or the like. In this case, the data extraction apparatus 100 defines, for each service registrant, extraction conditions that are satisfied by article data to be sent to the registrant. Article data is input to the extraction device 100 as needed, and is output to a corresponding file in accordance with the extraction conditions satisfied by the article data. The article data output to the file is periodically distributed to service registrants. Addition, deletion, or change of request of service registrants can be handled by adding or deleting extraction conditions or changing contents.

  FIG. 24 shows an example in which the data extraction device 100 is used in a highway usage survey system. In this case, data is input to the data extraction device 100 from the highway monitor system as needed. The extraction apparatus 100 defines extraction conditions for extracting only necessary data. Thereby, the extraction apparatus 100 sorts (filters) data according to the extraction conditions. The selected data is collated with master data by a coupler and developed into more detailed data. In the example, the company name “XX Transport” is added to the data of the car number “k 2104”. Data collated with the master data is aggregated and output by the aggregator, for example, for each company.

  In this embodiment, the data itself that distributes the output destination according to the extraction condition is input from the outside, but the data may be used for generation of data that is actually distributed or for identification. That is, it may be like encoded compressed data. Such data input may be performed by recording on the recording medium MD.

Claims (1)

A program to be executed by a computer to realize a data extraction device capable of extracting data satisfying a specified extraction condition from among obtainable data,
A function of acquiring the data;
A function of inputting the extraction condition;
A function of extracting data for each extraction condition by using one or more input extraction conditions according to the input function;
A function of outputting the data extracted for each extraction condition to the different output destinations by the function of extracting;
A program to realize